Monitoring users and conditions in a structure

ABSTRACT

Methods and apparatus include those used to monitor a position of a user in or proximate to a structure and the conditions in the structure while the user is present. Methods and apparatus may further orient the user in the structure to guide the user to locations or items of interest and using a virtual model, the user&#39;s smart device may provide information about the structure to the user. Precise locations of the user, conditions measured, and items therein may be quantified via simplified X, Y and Z coordinate determination and/or a determined direction of interest.

CROSS REFERENCE TO RELATED APPLICATIONS

The present application claims priority to U.S. patent application Ser.No. 16/165,517 dated Oct. 19, 2018, which claims priority from NonProvisional patent application Ser. No. 16/142,275, filed Sep. 26, 2018,and entitled METHODS AND APPARATUS FOR ORIENTEERING as a Continuation inPart Application: and to Non Provisional patent application Ser. No.16/161,823, filed Oct. 16, 2018 and entitled BUILDING MODEL WITH CAPTUREOF AS BUILT FEATURES AND EXPERIENTIAL DATA as a Continuation in PartApplication: and to Non Provisional patent application Ser. No.15/887,637, filed Feb. 2, 2018 and entitled BUILDING MODEL WITH CAPTUREOF AS BUILT FEATURES AND EXPERIENTIAL DATA as a Continuation in PartApplication: and to Non Provisional patent application Ser. No.15/703,310, filed Sep. 13, 2017 and entitled BUILDING MODEL WITH VIRTUALCAPTURE OF AS BUILT FEATURES AND OBJECTIVE PERFORMANCE TRACKING as aContinuation in Part Application: and to Non Provisional patentapplication Ser. No. 15/716,133, filed Sep. 26, 2017 and entitledBUILDING MODEL WITH VIRTUAL CAPTURE OF AS BUILT FEATURES AND OBJECTIVEPERFORMANCE TRACKING as a Continuation in Part Application: which claimspriority to Provisional Patent Application Ser. No. 62/712,714, filedJul. 31, 2018 and entitled BUILDING MODEL WITH AUTOMATED WOOD DESTROYINGORGANISM DETECTION AND MODELING: which claims priority to ProvisionalPatent Application Ser. No. 62/462,347, filed Feb. 22, 2017 and entitledVIRTUAL DESIGN, MODELING AND OPERATIONAL MONITORING SYSTEM: which claimspriority to Provisional Patent Application Ser. No. 62/531,955, filedJul. 13, 2017 and entitled BUILDING MODELING WITH VIRTUAL CAPTURE OF ASBUILT FEATURES; which claims priority to Provisional Patent ApplicationSer. No. 62/531,975 filed Jul. 13, 2017 and entitled BUILDINGMAINTENANCE AND UPDATES WITH VIRTUAL CAPTURE OF AS BUILT FEATURES as acontinuation in part application; the contents of each of which arerelied upon and incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates to apparatus and methods for aggregatingobjective measurements quantifying as built conditions and experientialconditions of a structure over time, and using those measurements todetermine, among other things, compliance with applicable buildingcodes. Precise locations of conditions measured are quantified viasimplified X, Y and Z coordinate determination in combination with adetermined direction of interest. More specifically, the presentinvention presents methods and apparatus for indicating a directionbased upon unique automated generation of a vector.

BACKGROUND OF THE INVENTION

Automated smart home systems are known. On a regular basis, additionalautomated controls of devices located within a home are developed,including: automated climate control; automated appliance control;automated lighting; automated security and the like. However, althoughmuch development has been made to provide a user with automated meansfor controlling an environment within the house or an appliance,heretofore, very little has been developed to assess and control theconditions integral to the structure of a house itself.

Moreover, it is very difficult to ascertain a location of a condition inrelation to specific features of a house structure, such as, a locationof a condition in relation to a kitchen or a bedroom, or a front door.

In addition, traditional methods of using automated design tools, suchas AutoDesk™, have been focused on the generation of a design plan foruse in construction of a facility, such as a processing plant. Anautomated design tool may be advantageous in the specifying of buildingaspects, materials and placement of features. Aspects may includebuilding features, such as walls, ingress/egress, utilities and evenequipment. However, usefulness of the design plan in taking concreteactions using, for example, a smart device, is also limited absent adirection of interest from any given point. While determining a positionon a coarse scale is known in the art, the required fine-scale positionand direction of interest are not.

Similarly, while traditional methods of using automated design tools,such as AutoDesk™, have greatly increased the capabilities of virtualmodels of facilities, very little has been done to quantify a deployedperformance of design features, such as equipment layout, capacity,throughout consumables walls, ingress/egress, windows, ceiling designs,textures, building materials, placement of structural beams, utilities,machinery location, machinery type, machinery capacity equipment.Accurate recreation of such design features in the field requires anindication of both location and direction.

More sophisticated design systems include “virtual reality” models.Virtual reality models may include two dimensional and/or threedimensional views from one or more user selected Vantage Points withinthe model of the structure. Virtual reality models also require adesignation of a Vantage Pont and a direction.

SUMMARY OF THE INVENTION

Accordingly, the present invention provides automated apparatus, devicesand methods of operation for quantifying vital conditions of a house andinfrastructure supporting the house within a property, or proximate to aproperty.

Generally speaking, a Global Positioning System (GPS) may be referencedto designate one or both of the First Geospatial Point and the SecondGeospatial Point in as much as GPS signals are available and theaccuracy afforded by the GPS is acceptable. In addition, a position maybe determined via other wireless reference mediums, such as WiFi,Bluetooth, ANT, Cell Tower signals, infrared beams or other mediums thatprovide wireless reference points.

The present invention provides for automated apparatus for improvedmodeling of construction, Deployment and updating of a Structure. Theimproved modeling is based upon generation of As Built and ExperientialData captured with one or both of Smart Devices and Sensors located inor proximate to the Structure. The automated apparatus is also operativeto model compliance with one or more performance levels for theStructure related to processing of a Product.

In another aspect of the present invention, a virtual model of aStructure extends beyond a design stage of the structure into an “AsBuilt” stage of the structure and additionally includes generation andanalysis of Experiential Data capturing conditions realized by theStructure during a Deployment stage of the structure.

In general, As Built and Experiential Data generated according to thepresent invention include one or more of: image data; measurements;component specifications of placement; solid state; electrical; andelectromechanical devices (or combination thereof); generate datacapturing conditions experienced by a structure. In addition, a user mayenter data, such as for example, data descriptive of an action taken bya service technician into an Augmented Virtual Mode (AVM). As Built andExperiential Data may be aggregated for a single structure or multiplestructures. Likewise, a Structure may comprise a single structure ormultiple structures.

As Built data is collected that quantifies details of how a specificphysical structure was actually constructed. According to the presentinvention, a Structure is designed and modeled in a 3D virtual setting.As Built data is combined with a design model in a virtual setting togenerate an AVM. As Built data may reflect one or more of: fabricationof the Structure; repair; maintenance; upgrades; improvements; and workorder execution associated with the Structure.

In addition, Experiential Data may be generated and entered into the AVMvirtual model of the structure. Experiential Data may include dataindicative of a factor that may be tracked and/or measured in relationto the Structure. Experiential data is typically generated by Sensors inor proximate to the Structure and may include, by way of non-limitingexample, one or more of: vibration sensors (such as piezo-electrodevices); accelerometers; force transducers; temperature sensingdevices; amp meters, ohmmeters, switches, motion detectors; lightwavelength capture (such as infrared temperature profile devices), waterflow meters; air flow meters; and the like. Some examples ofExperiential Data may include: details of operation of equipment ormachinery in the Structure; vibration measurements; electrical currentdraws; machine run times, machine run rates, machine run parameters;interior and/or exterior temperatures; opening and closings of doors andwindows; weight loads; preventive maintenance; cleaning cycles; aircirculation; mold contents; thermal profiles and the like. Automatedapparatus captures empirical data during construction of the Structureand during Deployment of the Structure.

By way of additional example, it may be determined that waterconsumption in a particular Structure, or a particular class ofprocessing plants, will be analyzed to determine if it is prudent tomake modifications to the particular Structure or class of Structures.The automated apparatus of the present invention will include As Builtdata for features of a structure that is accessed while modelingproposed modifications and upgrades. Relevant As Built Features mayinclude features for which relevancy may seem obvious, such as, forexample, one or more of: utility requirements, electrical, chemicalsupply, chemical waste disposal, air handling equipment, hoods, exhaustand filtering; plumbing; machinery models and efficiency. In addition,other As Built Features, for which relevancy may not seem obvious, butwhich unstructured queries draw a correlation may also be included.

Location of appliances, equipment, machines and utilities relative toother appliances, equipment, machines and utilities may also be deemedrelevant by unstructured query analysis. An unstructured query ofcaptured data quantifying actual chemical, atmosphere and water usagemay determine that certain configurations better meet an objective thanothers. For example, it may later be determined that the single storystructure is more likely to have a consistent internal temperature,lighting, ambient particulate or other trends than a multi-storystructure.

As discussed more fully below, captured data may include empiricalquantifications of a number of times a piece of machinery cycles on andoff, vibrations within a structure, temperature within a structure,doors opening and closing, quantity of products processed, hours ofoccupancy of the structure and other variable values. Captured data mayalso be used to generate a determination of how a structure is beingused, such as production cycles, quality, yield, rates, volumes, etc. Asdiscussed more fully below, empirical Sensor data associated with howparticular personnel behaves within a Structure may also be correlatedwith structure Performance based upon who occupies a particularstructure, when they occupy and for how long.

The automated apparatus combines a model of a structure that has beendesigned and provides precise additions to the model based upon datacapture of features actually built into the structure. This allows forservice calls that may include one or more of: repairs, upgrades,modifications and additions (hereinafter generally referred to as“Service Call”), may access data indicating an AVM combined with precisefeatures included in a building represented by As Built data, as well asExperiential Data and technical support for the features, maintenancelogs and schedules, “how to” documentation and video support, virtualconnection to specialists and experts, and a time line of original AsBuilt details and subsequent modifications. Modifications may includerepairs, updates and/or additions to a structure.

The improved methods taught herein provide for the performance ofrepairs, maintenance and upgrades via access to a system thatincorporates “As Built” data into the AVM. Geolocation and directionwill be used to access virtual reality representations of a structureincluding actual “As Built Imagery” incorporated into the AVM thataccurately indicates locations and types of features and also providesimages or other captured data. Exemplary data may include As Builtlocations of structural components (beams, headers, doorways, windows,rafters etc.); HVAC, electrical, plumbing, machinery, equipment, etc.Virtual repair may include “how to” instructions and video, technicalpublications, visual models comprised of aggregated data of similarrepair orders and the like. An onsite technician may verify correctlocation of an equipment unit based upon GPS, triangulation, directiondeterminations.

An AVM may additionally include virtual operation of equipment and useof modeled structure based upon aggregated data from one or more AsBuilt structures. Upon conclusion of a repair, maintenance, upgrade oraddition. Additional information quantifying time, place, nature ofprocedure, parts installed, equipment, new component location etc. maybe captured and incorporated into a virtual model.

Some embodiments of the present invention include capturing data ofprocedures conducted during preventive maintenance and/or a Service Calland inclusion of relevant data into a virtual model. Precise datacapture during a Service Call or during construction may include actuallocations of building features such as, electrical wiring andcomponents, plumbing, joists, headers, beams and other structuralcomponents, as well as other Sensor measurements. Data capture may beongoing over time as the building is used and modified, or updatedduring the life of a structure (sometimes referred to herein as the“Operational” or “Deployed” stage of the structure).

An Operational Stage may include, for example: occupation and use of aProperty, as well as subsequent modifications, repairs and structureimprovements. The Property may include one or more modeled structures,such as: a factory, processing plant, fabrication facility, server farm,power generator facility, an outbuilding and facilities included in aProperty. Smart Devices with unique methods of determining a locationand direction of data capture are utilized to gather data duringconstruction of modeled buildings or other structures and duringDeployment of the structures during the Operational Stage.

In general, Smart Devices provide ongoing collection of “As Built” and“Deployed” data that is captured during construction and Deployment of abuilding. The collected data is further correlated with design data andused to track Performance of features included in a design of processplants and/or features included within the confines of a Property parcel(“Property”).

In another aspect, collected data may be used to predict Performance ofa Property based upon features built into the structure and conditionsexperienced by the Property. As Built data may include modifications toa Property that are made during a construction phase, and/or during aDeployment phase, of a Property life cycle. Similarly, as Deployed datamay include details quantifying one or more of: machine operators,production quantity, yield, quality level, usage, maintenance, repairsand improvements performed on the Property.

In still another aspect of the present invention, predictive analyticsmay be performed to predict a life of various components included in theProperty. Maintenance procedures and replacement of consumables or otherparts may also be budgeted and scheduled based upon a correlation of a)design data; b) As Built data; and c) as used data. In addition,contemplated improvements may be modeled according to an expected returnon investment (“ROI”). An expected ROI may be calculated according toone or more of: an objective level of measurements, an amount of afungible item, such as kilowatt, gallon, currency, volume or otherquantity expended during the life of Deployment; and satisfaction ofusers and Performance.

Predictive analytics may include monitoring use of equipment andmachinery. The monitoring may include data collection that is stored ina controller and analyzed, such as, via artificial intelligenceroutines. In some embodiments, data gathered during monitoring may betransmitted to a centralized location and aggregated with other similartype buildings, building support equipment (e.g., HVAC, plumbing,electric) and appliances. Analytic profiles may be generated. PredictedPerformance and failures may be generated and used to schedule ServiceCalls before a physical failure occurs. Profiles may include degrees ofusage, consumables, electric current draws, vibration, noise, imagecapture and the like.

Still another aspect includes generation of virtual reality userinterfaces accessing the AVM based upon a) design data; b) As Builtdata; c) as used data; and d) improvement data. A virtual reality userinterface may be accessed as part of one or more of: a maintenanceroutine; to support a change order for the Property; and to contemplateimprovements in a Property. As Built and as deployed data may includedata quantifying repairs and updates to the Property.

In some embodiments, a) design data; b) As Built data; c) ExperientialData; and d) Lead Actions and Lag Benefit measurements, as they relateto multiple Properties may be aggregated and accessed to support one ormore Properties. Access to aggregated data may include execution ofartificial intelligence (AI) routines. AI routines may include, by wayof non-limiting example; structured algorithms and unstructured queriesoperative to predict Performance metrics and maintenance needs. AIroutines may access both initial designs and data aggregated duringbuild and deployment stages of the Property.

The details of one or more examples of the invention are set forth inthe accompanying drawings and the description below. The accompanyingdrawings that are incorporated in and constitute a part of thisspecification illustrate several examples of the invention and, togetherwith the description, serve to explain the principles of the invention:other features, objects, and advantages of the invention will beapparent from the description, drawings, and claims herein.

DESCRIPTION OF THE DRAWINGS

The accompanying drawings, that are incorporated in and constitute apart of this specification, illustrate several embodiments of theinvention and, together with the description, serve to explain theprinciples of the invention:

FIG. 1A illustrates a block diagram of inter-relating functions includedin automated systems according to the present invention.

FIG. 1B illustrates geolocation aspects that may be used to identify aProperty and corresponding data and predictions.

FIG. 1C illustrates a block diagram of ongoing data capture via SmartDevices and Sensors and support for predictive modeling based upon thesmart data capture.

FIG. 1D illustrates an exemplary Progressive Facility layout withvarious equipment delineated in a top-down representation according tosome embodiments of the present invention.

FIG. 1E illustrates a diagram of a user and directional image data.

FIG. 2 illustrates a block diagram of an Augmented Virtual Modelingsystem.

FIGS. 3A-3F illustrate exemplary aspects of collecting and displayingdata of a Structure generated during construction of the Structure.

FIG. 3G illustrates an exemplary key component of the model system, witha Performance monitor providing data via a communication system to themodel system.

FIG. 3H illustrates an exemplary virtual reality display in concert withthe present invention.

FIGS. 4A, 4B, and 4C illustrate an exemplary method flow diagrams withsteps relating to processes.

FIG. 5 illustrates location and positioning devices associated within aStructure.

FIG. 6 illustrates apparatus that may be used to implement aspects ofthe present invention including executable software.

FIG. 7 illustrates an exemplary handheld device that may be used toimplement aspects of the present invention including executablesoftware.

FIG. 8 illustrates method steps that may be implemented according tosome aspects of the present invention.

FIGS. 9A-D illustrates views of an AVM via a wearable eye displayaccording to some aspects of the present invention.

FIGS. 10A-C illustrates viewing areas of an AVM according to someaspects of the present invention.

FIGS. 11A-C illustrates vertical changes in an AVM viewable areaaccording to some aspects of the present invention.

FIG. 12 illustrates designation of a direction according to some aspectsof the present invention.

FIGS. 13-13C illustrate a device and vectors according to variousembodiments of the present invention.

FIG. 14 illustrates a vehicle acting as platform 1400 for supportingwireless position devices.

FIGS. 15A-15C illustrate movement of a smart device to generate a vectorand/or a ray.

FIG. 16 illustrates method steps that may be executed in practicing someembodiments of the present invention.

FIGS. 17A-B illustrates method steps that may be implemented in someembodiments of the present disclosure.

FIG. 18 demonstrates tables with exemplary sensor readings.

FIG. 19 illustrates an exemplary apparatus for attaching an exemplaryaccelerometer to a component of a Structure.

DETAILED DESCRIPTION

The present invention relates to methods and apparatus for improvedmodeling, Deployment and updating of a Structure based upon As Built andExperiential Data. As Built and Experiential Data may quantify anallocation of resources required for a level of Structure Performanceduring Deployment of the facility. Design, repair, maintenance andupgrades to a Structure are modeled with an automated system thatincorporates “As Built” data and “Experiential” data into a virtualmodel of the structure to determine a level of performance of theStructure.

The present invention provides automated apparatus and methods forgenerating improved Augmented Virtual Models (sometimes referred toherein as an “AVM”) of a Structure; the improved AVMs are capable ofcalculating a likelihood of achieving stated Performance Level specifiedby a user. In addition, the improved model may be operative to generatetarget Performance Metrics based upon As Built and Experiential Data.

The Augmented Virtual Model of the Property may include a conceptualmodel and progress through one or more of: a) a design stage; b) a buildstage; c) a Deployment stage; d) a service stage; e) a modificationstage; and f) a dispensing stage. As discussed more fully herein, an AVMaccording to the present invention may include original design datamatched to As Built data, which is captured via highly accurategeolocation, direction and elevation determination. As Built data ismatched with a time and date of data acquisition and presented intwo-dimensional (2D) and three-dimensional (3D) visual representationsof the Property. The augmented models additionally include data relatingto features specified in a Property design and data collected duringbuilding, Deployment, maintenance and modifications to the Property. Insome embodiments, a fourth dimension of time may also be included.

An Augmented Virtual Model includes a three- or four-dimensional modelin a virtual environment that exists parallel to physical embodimentsmodeled in the Augmented Virtual Model. Details of one or more physicalstructures and other features within a real estate parcel are generatedand quantified and represented in the Augmented Virtual Model. TheAugmented Virtual Model exists in parallel to a physical structure: theAVM includes virtual representations of physical structures andadditionally receives and aggregates data relevant to the structuresover time. The aggregation of data may be one or more of: a) accordingto an episode (e.g., onsite inspection, repair, improvement etc.); b)periodic; and c) in real time (without built-in delay).

The experience of the physical structure is duplicated in the virtualAVM. The Augmented Virtual Model may commence via an electronic modelgenerated using traditional CAD software or other design-type software.In addition, the AVM may be based upon values for variables, includingone or more of: usage of a Structure; usage of components within theStructure; environmental factors encountered during a build stage orDeployment stage; and metrics related to Performance of the Structure.The metrics may be determined, for example, via measurements performedby Sensors located in and proximate to structures located on theProperty or other Structures.

In another aspect, an Augmented Virtual Model may be accessed inrelation to modeling achievement of a stated Performance Level. Accuratecapture of As Built Features and aggregated data of similar buildings,equipment types, machinery and usage profiles assist in one or more of:predicting Performance Level, Yield, Quality, Volume of Production,selecting appropriate technicians to deploy to a Service Call; providingcorrect consumables and replacement parts, scheduling a preventativemaintenance; scheduling building, equipment and/or machinery upgrades;matching a building, equipment and machinery combination of a particulartype of Deployment; providing on site guidance during the Service Call;providing documentation relevant to the building, equipment andmachinery; providing access to remote experts that guide onsitetechnicians.

In some embodiments, a technical library specific to a particularproperty and location within the Property may be maintained for eachProperty and made accessible to an onsite technician and/or remoteexpert. The library may include, but is not limited to: structure,equipment/machinery manuals; repair bulletins, and repair/maintenance.Appropriate how-to videos may also be made available based upon an AVMwith As Built and Experiential Data.

In another aspect, a parts-ordering function may be included in theAugmented Virtual Model. Augmented parts ordering may allow a technicianto view an ordered part and view a virtual demonstration of the part inuse and procedures for replacing the part.

Aspects of the Augmented Virtual Model may be presented via a userinterface that may display on a tablet or other flat screen, or in someembodiments be presented in a virtual reality environment, such as via avirtual reality headset.

The present invention additionally provides for an Augmented VirtualModel to forecast Future Performance of a Property based upon the valuesof variables included in data aggregated during the design, build andDeployment of the Property sometimes referred to herein as: a) Designfeatures; b) As Built data; and c) as Deployed data.

The improved modeling system incorporates “As Built” data into theimproved design model. Subsequently, an onsite or remote technician mayaccess the As Built data to facilitate. The As Built data is generatedand/or captured via highly accurate geolocation, direction and elevationdetermination. Based upon the geolocation, direction and elevationdetermination, As Built data is incorporated into a design model at aprecise location within the AVM. In some embodiments, a time and date ofdata acquisition may be associated with updates to aspects of theimproved AVM such that a chronology of changes exists within the AVM.

Original design aspects and updated design aspects may be presented in2D and 3D visual representations of the Property. The present inventionprovides for systematic updates to As Built data during a Deployment ofthe Property. Updated data may verify and/or correct previously includeddata and also be used to memorialize modifications made during a ServiceCall or modification to a Property.

Some exemplary embodiments may include updates to an AVM that include,one or more of: quantifying a make and model of equipment and machineryon site; time and date notation of change in location specific data;Model accessed and/or updated according to X, Y, and Z coordinates anddistance data; X-axis, Y-axis data may include high level locationdesignation within the street address via triangulation (e.g., a streetaddress) and highly specific position designation (e.g., particular roomand wall); combination of two types of position data; GPS, DifferentialGPS; references used during triangulation; aggregate data acrossmultiple structures for reference; designs that perform well; designsthat fail; popularity of various aspects; access to and/or generationof, multiple Augmented Virtual Models; original and modified modelversions; index according to date/time stamp; index according tofeature; index according to popularity; index according to cost; indexaccording to User specific query; plumbing; electrical; HVAC; chemical,raw material, structural; access areas (i.e. crawl spaces, attics);periodic data and position capture with camera/Sensor attached to afixed position; and during one or more of: repair/maintenance/updates.

Accordingly, actual “As Built” imagery and location data is incorporatedinto the design model to accurately indicate a location and type offeature included in a structure, and provides “pictures” or othercaptured data. Exemplary data may include As Built locations ofstructural components (beams, headers, doorways, windows, rafters etc.);HVAC, electrical, plumbing, machinery, equipment, etc. A virtual realitymodel may additionally include virtual operation of machinery andequipment and use of a Structure based upon aggregated data from thestructure, as well as annotations and technical specifications relatingto features included in the As Built model of a Structure identified bytime, date, geolocation and direction.

In some embodiments, an initial digital model may be generated accordingto known practices in the industry. However, unlike previously knownpractices, the present invention associates an initial digital modelwith a unique identifier that is logically linked to a geolocation andone or both of date and time designation, and provides updates to theoriginal model based upon data captured at the geolocation during arecorded timeframe. In this manner, an AVM is generated that logicallylinks a digital model to a specific geographic location and actual AsBuilt data at the specific geographic location. The updated model may bevirtually accessed from multiple locations such as a field office,onsite, a technical expert, a financial institution, or other interestedparty.

In some preferred embodiments, the geographic location will be providedwith accurately placed location reference points. The location referencepoints may be accessed during activities involved in a Service Call onthe Property, such as a repair or upgrade to a structure or otherstructures included within a property parcel surrounding the structure.Accuracy of the reference points may or may not be associated withlocation relevance beyond the Property, however they do maintainaccuracy within the Property.

Preferred embodiments may also include reference points accuratelyplaced within a Structure located on the Property. As further discussedbelow, the reference points may include, by way of non-limiting example,a wireless transmission data transmitter operative to transmit anidentifier and location data; a visual identifier, such as a hash code,bar code, color code or the like; an infrared transmitter; a reflectivesurface, such as a mirror; or other means capable of providing areference point to be utilized in a triangulation process thatcalculates a precise location within the structure or other structure.

Highly accurate location position may be determined via automatedapparatus and multiple levels of increasingly accurate locationdetermination. A first level may include use of a GPS device providing areading to first identify a Property. A second level may use positiontransmitters located within, or proximate to, the Property to executetriangulation processes in view of on-site location references. A GPSlocation may additionally be associated with a high level generaldescription of a property, such as, one or more of: an address, a unitnumber, a lot number, a taxmap number, a county designation, Plattenumber or other designator. On-site location references may include oneor more of: near field radio communication beacons at known X-Y positionreference points; line of sight with physical reference markers; codedvia ID such as bar code, hash code, and alphanumeric or otheridentifier. In some embodiments, triangulation may calculate a positionwithin a boundary created by the reference points, which position isaccurate on the order of millimeters. In some embodiments, DifferentialGPS may be used to accurately determine a location of a Smart Devicewith a sub-centimeter accuracy.

In addition to a position determination, such as latitude and longitude,or other Cartesian Coordinate (which may sometimes be indicated as an “Xand Y” coordinate) or GPS coordinate, the present invention provides fora direction (sometimes referred to herein as a “Z” direction andelevation) of a feature for which As Built data is captured and importedinto the AVM.

According to the present invention, a direction dimension may be basedupon a movement of a device. For example, a device with a controller andan accelerometer, such as mobile Smart Device, may include a userdisplay that allows a direction to be indicated by movement of thedevice from a determined location acting as a base position towards anAs Built feature in an extended position. In some implementations, theSmart Device may first determine a first position based upontriangulation with the reference points and a second position (extendedposition) also based upon triangulation with the reference points. Theprocess of determination of a position based upon triangulation with thereference points may be accomplished, for example via executablesoftware interacting with the controller in the Smart Device, such as,for example via running an app on the Smart Devices.

In combination with, or in place of directional movement of a deviceutilized to quantify a direction of interest to a user, some embodimentsmay include an electronic and/or magnetic directional indicator that maybe aligned by a user in a direction of interest. Alignment may include,for example, pointing a specified side of a device, or pointing an arrowor other symbol displayed upon a user interface on the device towards adirection of interest.

In a similar fashion, triangulation may be utilized to determine arelative elevation of the Smart Device as compared to a referenceelevation of the reference points.

It should be noted that although a Smart Device is generally operated bya human user, some embodiments of the present invention include acontroller, accelerometer, data storage medium, Image Capture Device,such as a Charge Coupled Device (“CCD”) capture device and/or aninfrared capture device being available in a handheld or unmannedvehicle.

An unmanned vehicle may include for example, an unmanned aerial vehicle(“UAV”) or ground level unit, such as a unit with wheels or tracks formobility and a radio control unit for communication.

In some embodiments, multiple unmanned vehicles may capture data in asynchronized fashion to add depth to the image capture and/or a three-and/or four-dimensional (over time) aspect to the captured data. In someimplementations, UAV position will be contained within a perimeter andthe perimeter will have multiple reference points to help each UAV (orother unmanned vehicle) determine a position in relation to staticfeatures of a building within which it is operating and also in relationto other unmanned vehicles. Still other aspects include unmannedvehicles that may not only capture data but also function to perform atask, such as paint a wall, drill a hole, cut along a defined path, orother function. As stated throughout this disclosure, the captured datamay be incorporated into the virtual model of a Structure.

In another aspect, captured data may be compared to a library of storeddata using image recognition software to ascertain and/or affirm aspecific location, elevation and direction of an image capture locationand proper alignment with the virtual model. Still other aspects mayinclude the use of a compass incorporated into a Smart Device.

In still other implementations, a line of sight from a Smart Device,whether user-operated or deployed in an unmanned vehicle, may be used toalign the Smart Device with physical reference markers and therebydetermine an X-Y position as well as a Z position. Electronic altitudemeasurement may also be used in place of, or to supplement, a knownaltitude of a nearby reference point. This may be particularly useful inthe case of availability of only a single reference point.

Reference points may be coded via identifiers, such as a UUID(Universally Unique Identifier), or other identification vehicle. Visualidentifiers may include a bar code, hash code, alphanumeric or othersymbol. Three dimensional markers may also be utilized.

By way of non-limiting example, on site data capture may includedesignation of an XYZ reference position and one or more of: imagecapture; infrared radiation capture; Temperature; Humidity; Airflow;Pressure/Tension; Electromagnetic reading; Radiation reading; Soundreadings (i.e. level of noise, sound pattern to ascertain equipmentrunning and/or state of disrepair), and other vibration or Sensorreadings (such as an accelerometer or transducer).

In some embodiments, vibration data may be used to profile use of thebuilding and/or equipment and machinery associated with the building.For example, vibration detection may be used to determine a machineoperation, including automated determination between proper operation ofa piece of equipment and/or machinery and faulty operation of theequipment and/or machinery. Accelerometers may first quantify facilityoperations and production speed and/or capacity during operations.Accelerometers may also detect less than optimal performance ofequipment and/or machinery. In some embodiments. AI may be used toanalyze and predict proper operation and/or equipment/machinery failurebased upon input factors, including vibration patterns captured.Vibrations may include a “signature” based upon machine type andlocation within a structure human related activity, such as, by way ofnon-limiting example: machine and foot traffic, physical activities,appliance operations, appliance failures, raised voices, alarms andalerts, loud music, running, dancing and the like, as well as a numberof machines and/or people in the building and a calculated weight andmobility of the people.

Vibration readings may also be used to quantify operation of appliancesand equipment associated with the building, such as HVAC, circulators,water pumps, washers, dryers, refrigerators, dishwashers and the like.Vibration data may be analyzed to generate profiles for properly runningequipment and equipment that may be faulty and/or failing. The improvedvirtual model of the present invention embodied as an AVM may beupdated, either periodically or on one off occasions, such as during aservice call or update call.

In some embodiments, a fourth dimension in addition to the three spatialdimensions will include date and time and allow for an historical viewof a life of a structure to be presented in the virtual model.Accordingly, in some embodiments, onsite cameras and/or Sensors may bedeployed and data may be gathered from the on-site cameras and Sensorseither periodically or upon command. Data gathered may be incorporatedinto the improved virtual model.

In still another aspect, the AVM may aggregate data across multipleProperties and buildings. The aggregated data may include conditionsexperienced by various buildings and mined or otherwise analyzed, suchas via artificial intelligence and unstructured queries. Accordingly,the AVM may quantify reasons relating to one or more of: how toreposition machines, route workflow or otherwise improve, designs thatwork well; designs that fail; popular aspects; generate multiple VirtualModels with various quantified features; original and modified modelversions and almost any combination thereof.

Although data may be gathered in various disparate and/or related ways,an aggregate of data may be quickly and readily accessed via thecreation of indexes. Accordingly, indexes may be according to one ormore of: date/time stamp; feature; popularity; cost; User-specificquery; Plumbing; Electrical; HVAC; Structural aspects; Access areas;Periodic data and position capture with camera/Sensor attached to afixed position; indexed according to events, such as duringconstruction, during modification, or during Deployment; airflow; HVAC;machinery; traffic flows during use of structure; audible measurementsfor noise levels; and almost any other aspect of captured data.

In another aspect, an Augmented Virtual Model may receive datadescriptive of generally static information, such as, one or more of:product specifications, building material specifications, productmanuals, and maintenance documentation.

Generally static information may be utilized within the AugmentedVirtual Model to calculate Performance of various aspects of a Property.Dynamic data that is captured during one of: a) design data; b) builddata; and c) deployed data, may be used to analyze actual Performance ofa Property and also used to update an Augmented Virtual Model andincrease the accuracy of additional predictions generated by theAugmented Virtual Model. Maintenance records and supportingdocumentation may also be archived and accessed via the AVM. A varietyof Sensors may monitor conditions associated with one or both of thestructure and the parcel. The Sensors and generated data may be used toextrapolate Performance expectations of various components included inthe Augmented Virtual Model. Sensor data may also be aggregated withSensor data from multiple Augmented Virtual Model models from multiplestructures and/or Properties and analyzed in order to track and/orpredict Performance of a structure or model going forward.

Glossary

“Agent” as used herein refers to a person or automation capable ofsupporting a Smart Device at a geospatial location relative to a GroundPlane.

“Ambient Data” as used herein refers to data and data streams capturedin an environment proximate to a Vantage Point and/or an equipment itemthat are not audio data or video data. Examples of Ambient Data include,but are not limited to Sensor perception of one or more of: temperature,humidity, particulate, chemical presence, gas presence, light,electromagnetic radiation, electrical power, moisture and mineralpresence.

“Analog Sensor” and “Digital Sensor” as used herein include a Sensoroperative to quantify a state in the physical world in an analogrepresentation.

“As Built” as used herein refers to details of a physical structureassociated with a specific location within the physical structure orparcel and empirical data captured in relation to the specific location.

“As Built Features” as used herein refers to a feature in a virtualmodel or AVM that is based at least in part upon empirical data capturedat or proximate to a correlating physical location of the feature.Examples of As Built Features include placement of structural componentssuch as a wall, doorway, window, plumbing, electrical utility, machineryand/or improvements to a parcel, such as a well, septic, electric orwater utility line, easement, berm, pond, wet land, retaining wall,driveway, right of way and the like.

“As Built Imagery” (Image Data) as used herein shall mean image datagenerated based upon a physical aspect.

“Augmented Virtual Model” (sometimes referred to herein as “AVM”) asused herein is a digital representation of a real property parcelincluding one or more three dimensional representations of physicalstructures suitable for use and As Built data captured descriptive ofthe real property parcel. An Augmented Virtual Model includes As BuiltFeatures of the structure and may include improvements and featurescontained within a Structure and is capable of being updated byExperiential data.

“Property” as used herein shall mean one or more real estate parcelssuitable for a deployed Structure that may be modeled in an AVM.

“Directional Indicator” as used herein shall mean a quantification of adirection generated via one or both of: analogue and digitalindications.

“Directional Image Data” as used herein refers to image data capturedfrom a Vantage Point with reference to a direction. Image data mayinclude video data.

“Directional Audio” as used herein refers to audio data captured from aVantage Point within or proximate to a Property and from a direction.

“Deployment” as used herein shall mean the placement of one or more of:a facility machinery and an equipment item into operation.

“Deployment Performance” as used herein shall mean one or both of:objective and subjective quantification of one or more of: facility,machinery, an equipment item operated, or habitation-related VitalStatistics, which may be depicted in an AVM.

“Design Feature” as used herein, shall mean a value for a variabledescriptive of a specific portion of a Property. A Design Feature mayinclude, for example, a size and shape of a structural element or otheraspect, such as a doorway, window or beam; a material to be used, anelectrical service, a plumbing aspect, a data service, placement ofelectrical and data outlets; a distance, a length, a number of steps; anincline; or other discernable value for a variable associated with astructure or Property feature.

“Digital Sensor” as used herein includes a Sensor operative to quantifya state in the physical world in a digital representation.

“Experiential Data” as used herein shall mean data captured on orproximate to a subject Structure descriptive of a condition realized bythe Structure. Experiential data is generated by one or more of: digitaland/or analog sensors, transducers, image capture devices, microphones,accelerometers, compasses and the like.

“Experiential Sensor Reading” as used herein shall mean a value of asensor output generated within or proximate to a subject Structuredescriptive of a condition realized by the Structure. An ExperientialSensor Reading may be generated by one or more of: digital and/or analogsensors, transducers, image capture devices, microphones,accelerometers, compasses and the like.

“Facility” includes, without limitation, a manufacturing plant, aprocessing plant, or a residential structure.

“Ground Plane” as used herein refers to horizontal plane from which adirection of interest may be projected.

“Image Capture Device” or “Scanner” as used herein refers to apparatusfor capturing digital or analog image data, an Image capture device maybe one or both of: a two-dimensional camera (sometimes referred to as“2D”) or a three-dimensional camera (sometimes referred to as “3D”). Insome examples an Image Capture Device includes a charged coupled device(“CCD”) camera. An Image Capture Device may also be capable of taking aseries of images in a short time interval and associating the imagestogether to create videos for use in four-dimensional model embodiments.

“Lag Benefit” as used herein shall mean a benefit derived from, or inrelation to a Lead Action.

“Lead Actions” as used herein shall mean an action performed on, in, orin relation to a Property to facilitate attachment of a PerformanceLevel.

“Performance” as used herein may include a metric of an action orquantity. Examples of Performance may include metrics of: number ofprocesses completed, energy efficiency; length of service; cost ofoperation; quantity of goods processed or manufacture; quality of goodsprocessed or manufacture; yield; and human resources required.

“Performance Level” as used herein shall mean one or both of a quantityof actions executed and a quality of actions.

“Ray” as used herein refers to a straight line including a startingpoint and extending indefinitely in a direction.

“Sensor” as used herein refers to one or more of a solid state,electromechanical, and mechanical device capable of transducing aphysical condition or property into an analogue or digitalrepresentation and/or metric.

“Smart Device” as used herein includes an electronic device including,or in logical communication with, a processor and digital storage andcapable of executing logical commands. By way of non-limiting example, aSmart Device may include a smartphone, a tablet, a virtual realityviewer, and an augmented reality viewer.

“Structure” as used herein shall mean a structure capable of receivingin a processing material and/or a consumable and outputting a product,or a structure capable of habitation.

“Total Resources” as used herein shall mean an aggregate of one or moretypes of resources expended over a time period.

“Transceiver” as used herein refers to an electronic device capable ofone or both of wirelessly transmitting and receiving data.

“Vantage Point” as used herein refers to a specified location which maybe an actual location within a physical facility or residentialstructure, or a virtual representation of the actual location within aphysical facility or residential structure.

“Vector” as used herein refers to a magnitude and a direction as may berepresented and/or modeled by a directed line segment with a length thatrepresents the magnitude and an orientation in space that represents thedirection.

“Virtual Structure” (“VPS”): as used herein shall mean a digitalrepresentation of a physical structure suitable for deployment for aspecified use. The Virtual Structure may include Design Features and AsBuilt Features. The Virtual Structure may be included as part of an AVM.

“Vital Condition” as used herein refers to a condition measurable via adevice located in or proximate to a structure, wherein a value of themeasured condition is useful to determine the structures ability to meetset of predetermined conditions.

“WiFi” as used herein shall mean a communications protocol with theindustrial, scientific and medical (“ISM”) radio bands within thefrequency range of 6.7 MHz (megahertz) and 250 GHz (Gigahertz).

Referring now to FIG. 1A, a block diagram is shown that illustratesvarious aspects of the present invention and interactions between therespective aspects. The present invention includes an Augmented VirtualModel 111 of a Structure that includes As Built Features. The generationand inclusion of As Built Features, based upon location anddirection-specific data capture, is discussed more fully below. Data maybe transmitted and received via one or both of digital and analogcommunications, such as via a wireless communication medium 117.

According to the present invention, one or more Deployment PerformanceMetrics 112 are entered into automated apparatus in logicalcommunication with the AVM 111. The Deployment Performance Metric 112may include a purpose to be achieved during Deployment of a modeledStructure. By way of non-limiting example, a Deployment PerformanceLevel may include one or more of: a production or quantity; quality;yield; scalability; a level of energy efficiency; a level of waterconsumption; mean time between failure for equipment included in theStructure; mean time between failure for machinery installed in thestructure; a threshold period of time between repairs on the Structure;a threshold period of time between upgrades of the Structure; a targetmarket value for a Property; a target lease or rental value for aProperty; a cost of financing for a Property; Total Cost of ownership ofa Property; Total Cost of Deployment of a Property or other quantifiableaspect.

In some embodiments, Deployment Performance Metrics may be related to afungible item, such as a measurement of energy (kWh of electricity,gallon of fuel oil, cubic foot of gas, etc.); man hours of work; trademedium (i.e., currency, bitcoin, stock, security, option etc.); parts ofmanufactures volume of material processed or other quantity. Relatingmultiple disparate Deployment Performance Metrics to a fungible itemallows disparate Performance Metrics to be compared for relative value.

Modeled Performance Levels 113 may also be entered into the automatedapparatus in logical communication with the AVM 111. The ModeledPerformance Levels 113 may include an appropriate level of Performanceof an aspect of the structure in the AVM affected by the DeploymentPerformance Metric 112. For example, a Performance Level 113 for energyefficiency for a structure modeled may include a threshold of kilowatthours of electricity consumed by the structure on a monthly basis.Similarly, a target market value or lease value may be a thresholdpecuniary amount. In some embodiments, the threshold pecuniary amountmay be measured according to a period of time, such as monthly oryearly.

Empirical Metrics Data 114 may be generated and entered into theautomated apparatus on an ongoing basis. The Empirical Metrics Data 114will relate to one or more of the Deployment Performance Metrics and maybe used to determine compliance with a Deployment Performance Leveland/or a Performance Levels. Empirical Metrics Data 114 may include, byway of non-limiting example, one or more of: a unit of energy; an unitof water; a number of service calls; a cost of maintenance; a cost ofupgrades; equipment details, design details, machinery details,identification of human resources deployed; identification oforganizations deployed; number of human resources; demographics of humanresources (e.g., age, gender, occupations, employment status, economicstatus, requiring assistance with basic living necessities; and thelike); percentage of time structure is occupied; purpose of occupancy(e.g., primary residence, secondary residence, short term rental, longterm lease, etc.); Sensor readings (as discussed more fully below); manhours required for structure repair/maintenance/upgrades; total currency(or other fungible pecuniary amount) expended on behalf of a structureor property.

In addition to Empirical Metrics Data 114, Lead Actions and expected LagBenefits 115 that may cause an effect on one or both of a DeploymentPerformance Level 112 and a Performance Level 113, may be entered intothe automated apparatus. A Lead Action may include an action expected toraise, maintain or lower an Empirical Metrics Data 114. For example, anaction to install water efficient plumbing fixtures may be scheduled inorder to improve water consumption metrics. Similar actions may relateto electrically efficient devices, or automatic electric switches beinginstalled; preventive maintenance being performed; structure automationdevices being installed and the like. Other Lead Actions may includelimiting a demographic of occupants of a structure to a certaindemographic, such as senior citizens. An expected benefit may bemeasured in Lag Benefit measurements, such as those described asEmpirical Metrics Data 114, or less tangible benefits, such as occupantsatisfaction.

The automated apparatus may also be operative to calculate FuturePerformance 116 based upon one or more of: AVM Model with As Built Data111; Deployment Performance Metrics 112; Modeled Performance Levels 113and Empirical Metrics Data 114. Future Performance may be calculated interms of an appropriate unit of measure for the aspect for whichPerformance is calculated, such as, for example: an energy unit; manhours; mean time between failures and dollar or other currency amount.

Calculation of Future Performance 116 may be particularly useful tocalculate Total Resources calculated to be required to support aparticular structure, group of structures, properties and/or group ofproperties over a term of years (“Total Resources Calculated”). TotalResources Calculated may therefore be related to calculations of FuturePerformance 116 and include, for example, one or more of: energy units;water units; man hours; equipment; machinery and dollars (or othercurrency or fungible item). In some embodiments, calculations of FuturePerformance may include a Total Cost of Ownership for a term of years.For example, a Total Cost of Ownership for a Property may include apurchase amount and amounts required for maintenance, repair andupgrades from day one of Deployment through twenty years of Deployment(a shorter or longer term of years may also be calculated).

Accordingly, some embodiments may include a calculation of TotalResources required that includes a purchase price of a property with aStructure, that incorporates a total cost associated with the propertyover a specified term of years. The total cost will be based upon theAVM with As Built Data 111; Deployment Performance Metrics 112; ModeledPerformance Levels 113 and Empirical Metrics Data 114.

Moreover, Total Resources required may be aggregated across multipleproperties and. Structures. Aggregation of properties may be organizedinto property pools to mitigate risk of anomalies in the Calculation ofFuture Performance. Of course the benefits of property ownership and/ormanagement may also be pooled and compared to the Total Resourcesrequired. In various embodiments, different aspects of calculated FuturePerformance 116 may be aggregated and allocated to disparate parties.For example, first aggregation may relate to man hours of techniciantime for structure repair and maintenance and the fulfillment ofobligations related to the aggregation may be allocated to a firstparty. A second aggregation may relate to machinery Performance andobligations allocated to a second party. A third aggregation may relateto equipment Performance and obligations allocated to a third party.Other aggregations may similarly be allocated to various parties. Insome embodiments, financial obligations incorporating one or both ofacquisition cost and ongoing Deployment costs may be allocated andfinanced as a single loan. Other embodiments include a calculated FuturePerformance cost being incorporated into a purchase price.

An important aspect of the present invention includes definition andexecution of Lead Actions based upon one or more of: the AVM Model withAs Built Data 111; Deployment Performance Metrics 112; ModeledPerformance Levels 113; Empirical Metrics Data 114 and Calculations ofFuture Performance 116.

Referring now to FIG. 1B, an AVM is generally associated with a Propertythat includes a real estate parcel 109-111, 109A-111A. According to someembodiments, one or more of an improvement, a repair, maintenance and anupgrade are performed on the Property. The Property is identifiedaccording to an automated determination of a location and a particularposition, elevation and direction are further determined automaticallywithin the Property. Smart Devices may be used to access data recordsstored in an AVM according to a unique identifier of a physical locationof the real estate parcel 109-111, 109A-111A.

As illustrated, a map of real estate parcels 109-111, 109A-111A is shownwith icons 110A-111A indicating parcels 110-111 that have virtualstructures 110A-111A included in a virtual model associated with theparcels. Other parcels 113 have an indicator 113A indicating that avirtual model is in process of completion.

In some methods utilized by the present invention, data in an AVM may beaccessed via increasingly more accurate determinations. A first level ofgeospatial location determinations may be based upon a real estateparcel 109-111, 109A-111A and a second geospatial determination may bemade according to position locators (discussed more fully below)included within the boundaries of the real estate parcel 109-111,109A-111A. Still more accurate location position may be calculatedaccording to one or both of a direction determination and anaccelerometer. The position may be calculated using an accelerometer byassuming a known initial position, and using known methods of numericalintegration to calculate displacement from said initial position.Accordingly, it is within the scope of the present invention to access arecord of a design model for a specific wall portion within a structurebased upon identification of a real estate parcel 109-111, 109A-111A anda location within a structure situated within the real estate parcel109-111, 109A-111A and height and direction. Likewise the presentinvention provides for accessing As Built data and the ability to submitAs Built data for a specific portion of a structure based upon anaccurate position and direction determination.

In some implementations of the present invention, a Property uniqueidentifier may be assigned by the AVM and adhere to a standard foruniversally unique identifiers (UUID); other unique identifiers may beadopted from, or be based upon, an acknowledged standard or value. Forexample, in some embodiments, a unique identifier may be based uponCartesian Coordinates, such as GPS coordinates. Other embodiments mayidentify a Property according to one or both of: a street address and atax map number assigned by a county government of other authority.

In some embodiments, an AVM may also be associated with a larger groupof properties, such as a manufacturing plant, research and development,assembly, a complex, or other defined arrangement.

As illustrated, in some preferred embodiments, an electronic recordcorrelating with a specific Property may be identified and then accessedbased upon coordinates generated by a GPS device, or other electroniclocation device. The GPS device may determine a location and correlatethe determined location with an AVM record listing model data, As Builtdata, improvement data, Performance data, maintenance data, cost ofoperation data, return on investment data and the like.

Referring now to FIG. 1C, a relational view of an Augmented VirtualModel 100 with a Virtual Structure 102B is illustrated, as well as aphysical structure 102A. The Augmented Virtual Model 100 includes avirtual model stored in digital form with a design aspect that allowsfor a physical structure 102A suitable for use to be designed andmodeled in a virtual environment. The design aspect may referencePerformance data of features to be included in a Virtual Structure 102Band also reference variables quantifying an intended use of the VirtualStructure 102B. The Virtual Structure 102B and the Augmented VirtualModel 100 may reside in a virtual setting via appropriate automatedapparatus 108. The automated apparatus 108 will typically include one ormore computer servers and automated processors as described more fullybelow and may be accessible via known networking protocols.

The Physical Structure 102A may include transceivers 120 or other typeof Sensor or transmitter or receivers that monitor an area of ingressand egress 122, such as a doorway, elevator and/or loading dock.Reference point transceivers 121 may be used as wireless references of ageospatial position. A wireless communication device 139 may also linklogical infrastructure within the structure 102A with a digitalcommunications network.

In correlation with the design aspect, the present invention includes anAs Built Model 101 that generates a Virtual Structure 102A in thecontext of the Augmented Virtual Model 100. The As Built Model 101includes virtual details based upon As Built data captured on orproximate to a physical site of a related physical structure 102A. TheAs Built data may be captured, for example, during construction ormodification of a physical structure 102A.

The As Built Model 101 may include detailed data including imagecaptures via one or more image capture devices 107 and physicalmeasurements of features included in the physical structure 102A. Thephysical measurements may be during a build phase of the physicalstructure; or subsequent to the build phase of the physical structure.In some embodiments, original As Built measurements may be supplementedwith additional data structure data associated with repairs orimprovements are made to the physical structure. Details of recordablebuild aspects are placed as digital data on a recordable medium 104included in the automated apparatus 108.

The digital data included on a recordable medium 104 may thereforeinclude, for example, one or more of: physical measurements capturingExperiential Data; image data (e.g., digital photos captured with a CCDdevice or an Image Capture Device); laser scans; infrared scans andother measurement mediums. One or more records on the recordable medium104 of an As Built structure may be incorporated into the AugmentedVirtual Model 100 thereby maintaining the parallel nature of theAugmented Virtual Model 100 with the physical structure 102A.

In some embodiments, As Built data on a recordable medium 104 may begenerated and/or captured via an image capture device 117.

As the physical structure is deployed for use, subsequent measurementsthat generate and/or capture Experiential Data may be made andincorporated into the Augmented Virtual Model 100. In addition, a usermay access and update 103 the Augmented Virtual Model 100 to ascertainfeatures of the physical structure 102A that have been virtuallyincorporated into the Augmented Virtual Model 100. In some examples, atablet, handheld network access device (such as, for example a mobilephone) or other device with automated location service may be used todetermine a general location of a physical structure 102A. For example,a smart phone with GPS capabilities may be used to determine a physicaladdress of a physical structure, such as 123 Main Street. Stored recordscontaining data relating to 123 Main Street may be accessed via theInternet or other distributed network.

In addition to the use of GPS to determine a location of a User Device,the present invention provides for a real estate parcel with a physicalstructure 102A that includes more radio frequency (or other mechanism)location identifiers 109. Location identifiers 109 may include, forexample, radio transmitters at a defined location that may be used toaccurately identify via triangulation, a position of a user device 106,such as a: tablet, smart phone or virtual reality device. The positionmay be determined via triangulation, single strength, time delaydetermination or other process. In some embodiments, triangulation maydetermine a location of a user device within millimeters of accuracy.

Other location identifiers may include, by way of non-limiting example,RFID chips, a visual markings (i.e. a hash or barcode), pins or otheraccurately placed indicators. Placement of the location identifiers maybe included in the AVM and referenced as the location of the physicaluser device is determined. As described above, specific locationidentifiers may be referenced in the context of GPS coordinates or othermore general location identifiers.

Based upon the calculated location of the user device 106, details ofthe physical structure 102A may be incorporated into the VirtualStructure 102B and presented to a user via a graphical user interface(GUI) on the user device 106.

For example, a user may approach a physical structure and activate anapp on a mobile user device 106. The app may cause the user device 106to activate a GPS circuit included in the user device and determine ageneral location of the user device 106, such as a street addressdesignation. The general location will allow a correct AVM 104B to beaccessed via a distributed network, such as the Internet. Once accessed,the app may additionally search for one or more location identifiers 109of a type and in a location recorded in the AVM. An AVM may indicatethat one or more RFID chips are accessible in a kitchen, a living roomand each bedroom of a structure. The user may activate appropriateSensors to read the RFID chips and determine their location. In anotheraspect, an Augmented Virtual Model 100 may indicate that locationidentifiers 109 are placed at two or more corners (or other placement)of a physical structure 102A and each of the location identifiers 109may include a transmitter with a defined location and at a definedheight. The user device 106, or other type of controller, may thentriangulate with the location identifiers 109 to calculate a preciselocation and height within the physical structure.

Similarly, a direction may be calculated via a prescribed movement ofthe user device 106 during execution of code that will record a changein position relative to the location identifiers 109. For example, auser smart device, such as a smart phone or user device 106 may bedirected towards a wall or other structure portion and upon execution ofexecutable code, the smart device may be moved in a generally tangentialdirection towards the wall. The change in direction of the user device106 relative to the location identifiers 109 may be used to calculate adirection. Based upon a recorded position within the structure 102A andthe calculated direction, a data record may be accessed in the AugmentedVirtual Model 100 and a specific portion of the Augmented Virtual Model100 and/or the Virtual Structure 102B may be presented on the userdevice 106. In other embodiments, a direction may be made, or verifiedvia a mechanism internal to the smart device, such as a compass oraccelerometer.

In still another aspect of the present invention, in some embodiments,transmissions from one or more location identifiers 109 may becontrolled via one or more of: encryption; encoding; passwordprotection; private/public key synchronization or other signal accessrestriction. Control of access to location identifiers 109 may be usefulin multiple respects, for example, a location identifier mayadditionally function to provide access to data, a distributed networkand/or the Internet.

The Virtual Structure 102B may include one or both of: historical dataand most current data relating to aspects viewable or proximate to theuser device 106 while the user device is at the calculated location inthe physical structure 102A. In this way, the parallel virtual world ofthe Augmented Virtual Model 100 and the Virtual Structure 102B maypresent data from the virtual world that emulates aspects in thephysical world, and may be useful to the user accessing the user device106, while the user device is at a particular physical location. Asdiscussed within this document, data presented via the Augmented VirtualModel 100 may include one or more of: design data, As Built data,Experiential Data, Performance data relating to machinery and/orfeatures of the Augmented Virtual Model 100 or physical structure;maintenance data, and annotations.

Annotations may include, for example, a user's or designer's noterecorded at a previous time (such as a Service Technician's notation), aservice bulletin, maintenance log, operation instructions or a personalnote to a subsequent user, such as a virtual “John Smith was here” suchguest log indicating who had frequented the location. Annotations mayinclude one or both of text and image data. For example, an annotationmay include an image of the location captured at a given time and date.The image may be of a personal nature, i.e. the living room while theSmith's owned the structure, or a professional nature, i.e. the livingroom after being painted by XYZ Contractor on a recorded date. In someembodiments, annotations may be used to indicate completion of a workorder. Recordation of completion of a work order may in turn trigger apayment mechanism for paying an entity contracted to complete the workorder. In another aspect, annotations may relate to an AVM or a VirtualStructure as a whole, or to a particular aspect that is proximate to alocation of the user device within the Virtual Structure.

In some embodiments, details of a proposed use of a structure and parcelmay be input into a design module and used to specify or recommendfeatures to be included in an Augmented Virtual Model 100.

According to the present invention, features of a Structure and parcelare generated within a digital design model and then tracked as thefeatures are implemented in a build process and further tracked inPerformance of the structure as it is placed into use. To the extentavailable, Performance is tracked in the context of variables relatingto use. Variables may include, for example: a use of the structure, suchas manufacturing, processing, or habitation; a number of resourcesaccessing in a structure; demographics of the human resources; number ofmonths per year the structure is deployed for use; which months of theyear a structure is deployed for use; which hours of the day thestructure is occupied and other relevant information.

As Experiential Sensor Readings are generated they may be memorializedto generate Experiential Data associated with a physical structure 102A.The Experiential Data is collected and analyzed via structured queriesand may also be analyzed with Artificial Intelligence processes such asunstructured queries to derive value. In some embodiments, ExperientialData may also be associated with a human and/or an animal interactingwith the structure 102A. Whereas former process plants were generallydesigned and built to mitigate against variability in a human 118 andbetween disparate humans 118. The present invention allows for humanvariability to be monitored via sensors 119 and the structure to bemodified to optimally inter-relate with the values for variablesattributable to a human 118 that will inhabit or otherwise interact withthe structure 102A. Human (and/or animal) may be quantified with sensors119 installed on or proximate to the Human 118. Alternatively, sensors117 located in, or proximate to, a structure 102A may be used to monitorhuman variability. Biosensors may be used to provide empirical data ofhumans 118 interacting with a structure may be analyzed using structuredor unstructured queries to device relationships between structureperformance and human biometrics. Accordingly, sensors may be used toquantify interaction between a human 118 and an As Built structure 102Aaccording to physiological and behavioral data, social interactions, andenvironmental factors within the structure, actions undertaken,movements, and almost any quantifiable aspect.

As Built Features and biometrics may be further utilized to controlvarious structure automation devices. Structure automation devices mayinclude, by way of non-limiting example one or more of: automated locksor other security devices; thermostats, lighting, heating, chemicalprocessing, cutting, molding, laser shaping, 3D printing, assembly,cleaning, packaging and the like. Accordingly, a structure with recordedAs Built design features and vibration sensors may track activities in astructure and determine that a first occupant associated with a firstvibration pattern of walking is in the structure. Recorded vibrationpatterns may indicate that person one is walking down a hallway andautomatically turn on appropriated lighting and adjust one or more of:temperature, sound and security. Security may include locking doors forwhich person one is not programmed to access. For example, a firstpattern of vibration may be used to automatically ascertain that aperson is traversing an area of a structure for which a high level ofsecurity is required or an area that is designated for limited accessdue to safety concerns. As Built data has been collected. Otherstructure automation may be similarly deployed according to As Builtdata, occupant profiles, biometric data, time of day, or othercombination of available sensor readings.

Referring now to FIG. 1D, according to the present invention a virtualmodel 120 is generated that correlates with a physical facility 102A andincludes virtual representations of As Built features and ExperientialData. As discussed more fully herein, the virtual model may include anAVM 111 with As Built data, such as image data and measurements,included within the model. In addition, sensor data may be collectedover time and incorporated into the AVM 111. The AVM 111 may includevirtual representations of one or more of: sensors 125; equipment126-128; controls 131; infrastructure 129, such as HVAC, utilities, suchas electric and water, gas lines, data lines, etc. and vantage points121.

In some implementations, a virtual reality headset may be worn by a userto provide an immersive experience from a vantage point 121 such thatthe user will experience a virtual representation of what it would belike to be located at the vantage point 121 within the facility 122 at aspecified point in time. The virtual representation may include acombination of Design Features, As Built Data and Experiential Data. Avirtual representation may therefore include a virtual representation ofimage data via the visual light spectrum, image data via infrared lightspectrum, noise and vibration reenactment. Although some specific typesof exemplary sensor data have been described, the descriptions are notmeant to be limiting unless specifically claimed as a limitation and itis within the scope of this invention to include a virtualrepresentation based upon other types of captured sensor data may alsobe included in the AVM 111 virtual reality representation.

Referring now to FIG. 1E, a user 131 is illustrated situated within anAVM 111. The user 131 will be virtually located at a Vantage Point 137and may receive data 136, including, but not limited to one or more of:image data 134, audio data 135 and Ambient Data 136. The user 131 mayalso be provided with controls 133. Controls 133 may include, forexample, zoom, volume, scroll of data fields and selection of datafields. Controls may be operated based upon an item of Equipment 132within a Field of View 138 of the User 131 located at a vantage point137 and viewing a selected direction (Z axis). The user is presentedwith Image Data from within the AVM 111 that includes As Built data andvirtual design data.

Additional examples may include sensor arrays, audio capture arrays andcamera arrays with multiple data collection angles that may be complete360 degree camera arrays or directional arrays, for example, in someexamples, a sensor array (including image capture sensors) may includeat least 120 degrees of data capture, additional examples include asensor array with at least 180 degrees of image capture; and still otherexamples include a sensor array with at least 270 degrees of imagecapture. In various examples, data capture may include sensors arrangedto capture image data in directions that are planar or oblique inrelation to one another.

Referring now to FIG. 2, a functional block illustrates variouscomponents of some implementations of the present invention. Accordingto the present invention automated apparatus included in the AVM 201 areused to generate a model of a Virtual Structure (“VPS”) and may alsoincorporate a model and associated real estate parcel (“VPS”). One ormore pieces of equipment that will be deployed in the Property may beincluded into the augmented virtual model 201, equipment may include,for example: machinery 222; building support items 212, and utilitiessupport 213. The AVM 201 may model operational levels 204 duringdeployment of a facility and associated machinery, equipment, furniture,or other fixtures included in the AVM 201. Machinery 211 may include,for example, manufacturing tools, robots or other automation, transporttools, chemical processing machine, physical processing machine,assembly machine, heat processing machine, cooling machine, depositiondevice, etching device, welding apparatus, cutting apparatus, formingtool, drilling tool, shaping tool, transport machine, structureautomation, air purification or filter systems, noise containment deviceand the like. Utility support equipment may include cabling, dishantennas, Wi-Fi, water softener, water filter, power, chemical supply,gas supply, compressed air supply and the like, as well as uptime anddowntime associated with a facility utility and uptime and down time 243of one or more aspects of the facility.

The AVM 201 calculates a predicted Performance of the AVM and generatesOperational Levels 204 based upon the Performance 222, wherein“Performance” may include one or more of: total cost of deployment 214;operational experience 203 which may include one or both of: objectiveempirical measurements and satisfaction of operator's use an As Builtphysical model based upon the AVM; operational expectations 204, totalmaintenance cost 206, and residual value of an As Built following a termof years of occupation and use of an As Built Facility based upon theAVM. Performance 221 may also be associated with a specific item ofmachinery 211.

In another aspect, actual Operational Experience 203 may be monitored,quantified and recorded by the AVM 201. Data quantifying the OperationalExperience 203 may be collected, by way of non-limiting example, fromone or more of: Sensors incorporated into an As Built structure;maintenance records; utility records indicating an amount of energy 202(electricity, gas, heating oil) consumed; water usage; periodicmeasurements of an As Built structure, such as an infra-red scan ofclimate containment, air flow through air handlers, water flow, waterquality and the like; user surveys and maintenance and replacementrecords.

In still another aspect, a warranty 205 covering one or both of partsand labor associated with an As Built structure may be tracked,including replacement materials 207. The warranty 205 may apply to anactual structure, or one or more of machinery 211; building support 212item; and utility support item 213.

The AVM 201 may take into account a proposed usage of a Deployment of aStructure based upon values for Deployment variables, and specifyaspects of one or more of: Machine s 211; building support 212; andutility support 213 based upon one or both of a proposed usage andvalues for Deployment variables. Proposed usage may include, forexample, how many human resources will occupy a Structure, demographicsof the resources that will occupy the Structure; percentage of time thatthe Structure will be occupied, whether the Structure is a primaryresidence, whether the Structure is a leased property and typicalduration of leases entered into, environmental conditions experienced bythe Structure, such as exposure to ocean salt, Winter conditions, desertconditions, high winds, heavy rain, high humidity, or other weatherconditions.

In another aspect, Deployment may relate to biometrics or other dataassociated with specific occupants of a structure. Accordingly, in someembodiments, sensors may monitor biologically related variables ofoccupants and/or proposed occupants. The biometric measurements may beused to determine one or both of Lead Actions and Lag Metrics. Leadactions may include one or more of: use of specific building materials,selection of design aspects; Deployment of structure equipment;Deployment of machinery; terms of a lease; length of a lease: terms of amaintenance contract; and structure automation controls.

According to the present invention, design aspects and structurematerials 210 may also be based upon the proposed usage and values forDeployment variables. For example, a thicker exterior wall with higherinsulation value may be based upon a structures location in an adverseenvironment. Accordingly, various demographic considerations andproposed usage of a structure may be used as input in specifying almostany aspect of a Structure.

Total Cost of Deployment (TCD)

In still another consideration, a monetary value for one or more of: aTotal Cost of Deployment (“TCD”). Total maintenance cost (“TMC”) and adesired return on investment (“ROI”) for a Property may be used as inputfor one or more design aspects included in an Augmented Virtual ModelSystem 200. Total Cost of Ownership, Total Maintenance Cost and ROI maybe used to determine optimal values of variables 202-205, 210-213specified in an Augmented Virtual Model System 200 and incorporated intoan As Built structure, and other improvements to a real estate parcel.

A Total Cost of Deployment 214 may change based upon a time period 215used to assess the Total Cost of Deployment 214. A ROI may include oneor more of: a rental value that may produce a revenue stream, a resalevalue, a cost of operation, real estate taxes based upon structurespecifications and almost any other factor that relates to one or bothof a cost and value.

Desirable efficiency and Performance may be calculated according to oneor more of: established metrics, measurement protocols and pastexperience. The AVM 201 and associated technology and software may beused to support a determination of a TCD. In another aspect, a TCD maybe based upon an assembly of multiple individual metrics, procedures toassess metrics, procedures to adjust and optimize metrics and proceduresto apply best results from benchmark operations. In the course ofmanaging Total Cost of Ownership, in some examples, initial steps mayinclude design aspects that model an optimal design based upon TotalCost of Ownership metrics and also model designed algorithms used toassess Total Cost of Ownership metrics.

In the following examples, various aspects of Total Cost of Deployment214, Total Maintenance Costs, and associated metrics, are considered inthe context of calculating a target Total Cost of Deployment 214.Accordingly, the AVM may be used to TCD optimization.

A designed Structure is ultimately built at a site on a real estateparcel. A build process may be specified and provide metrics that may beused in a process designed by a AVM 201 and also used as a physicalbuild proceeds. In some examples, time factors associated with aphysical build may be important, and in some examples time factorsassociated with a physical build may be estimated, measured and actedupon as they are generated in a physical build process. Examples of timefactors may include, one or more of: a time to develop and approve siteplans; a time to prepare the site and locate community providedutilities or site provided utilities; a time to lay foundations; a timeto build structure; a time to finish structure; a time to installinternal utilities and facilities related aspects; a time to install,debug, qualify and release equipment; times to start production runs andto certify compliance of production are all examples of times that canbe measured by various techniques and sensing equipment on a Structure'ssite. Various time factors for a build are valuable and may becomeincreasingly valuable as a physical build proceeds since the monetaryinvestment in the project builds before revenue flows and monetaryinvestments have clearly defined cost of capital aspects that scale withthe time value of money.

Various build steps may include material flows of various types.Material flow aspects may be tracked and controlled for cost andefficiency. Various materials may lower a build materials cost, butraise time factors to complete the build. Logical variations may becalculated and assessed in an AVM 201 and optimal build steps may begenerated and/or selected based upon a significance placed upon variousbenefits and consequences of a given variable value. Physical buildmeasurements and/or sensing on physical build projects may also be usedas input in an assessment of economic trade-offs.

The equipment deployed may incur a majority of a build cost dependingupon user defined target values. The AVM may model and presentalternatives including one or more of: cost versus efficiency, quality240, time to build, life expectancy, market valuation over time. A costto build may be correlated with cost to deploy and eventual resale. Anoverall model of a Total Cost of Deployment 214 may include any or allsuch aspects and may also include external. In some examples, the natureof equipment trade-offs may be static and estimations may be made fromprevious results. In some other examples, changes in technology,strategic changes in sourcing, times of acquisition and the like mayplay into models of Total Cost of Deployment 214.

In some examples, an initial efficiency of design which incurs largecosts at early stages of a project may have a dominant impact on TotalCost of Deployment 214 when time factors are weighted to real costs. Inother examples, the ability of a Structure to be flexible over time andto be changed in such flexible manners, where such changes areefficiently designed may dominate even if the initial cost aspects maybe less efficient due to the need to design in flexibility. As aStructure is built, and as it is operated the nature of changingcustomer needs may create dynamic aspects to estimations of Total Costof Deployment 214. Therefore, in some examples, estimates on theexpected dynamic nature of demands on a Structure may be modeled againstthe cost aspects of flexibility to model expectations of Total Cost ofDeployment 214 given a level of change.

In some examples, factors that may be less dependent on extrinsicfactors, such as product demand and the like may still be importantmetrics in Total Cost of Deployment 214. Included in the As Builtfactors may be calculations such as HVAC temperature load, in whichpersonnel and seasonal weather implications may be important. AVM modelsmay include a user interface to receive value useful in the AVM models.In addition, electronic monitoring, via Sensors that may determineenergy consumption, includes for example: electricity, fuel oil, naturalgas, propane and the like may be useful for estimation and measurement.

Temperatures may be monitored by thermocouples, semiconductor junctionbased devices or other such direct measurement techniques. In otherexamples, temperature and heat flows may be estimated based on photonbased measurement, such as surveying the Structure with infra-redimaging or the like.

Utility load may be monitored on a Structure wide basis and/or at pointof use monitoring equipment located at hubs or individual pieces ofequipment itself. Flow meters may be inline, or external to pipes wiresor conduits. Gases and liquid flows may be measured with physical flowmeasurements or sound based measurement. In other examples, electricitymay be monitored as direct current measurements or inferred inductivecurrent measurement.

In some examples, the nature and design of standard usage patterns of aStructure and an associated environment may have relevance to Total Costof Ownership. For example, usage that includes a larger number ofingress and egress will expose an HVAC system to increased load andusage that includes a significant number of waking hours withinhabitants in the building may incur increased usage of one or more of:machinery 211; building support devices 212; and utilities 234.

The nature and measurement aspects of vibration in the Structure mayalso be modeled and designed as the Structure is built. There may benumerous means to measure vibrations from capacitive and resistive basedmeasurements to optical based measurements that measure a subtle changein distance scale as a means of detecting vibration. Vibration mayresult from a Structure being located proximate to a roadway, train,subway, airport, tidal flow or other significant source of relativelyconsistent vibration. Vibration may also be more periodic, such asearthquake activity. In still another aspect, vibration may result fromhuman traffic within the Property. The use of vibration monitoringSensors may indicate various activities that take place within thestructure and facilitate more accurate modeling of a life expectancy ofvarious aspects of the structure as well as machines located within thestructure.

Noise levels are another type of vibrational measurement which isfocused on transmission through the atmosphere of the Structure. In somecases, noise may emanate from one location after moving through solidstructure from its true source at another location. Thus, measurement ofambient sound with directional microphones or other microphonic sensingtypes may be used to elucidate the nature and location of noiseemanations. In some cases, other study of the noise emanations may leadto establishment of vibrational measurement of different sources ofnoise. Floors, ceilings, doorways, countertops, windows and otheraspects of a Structure may be monitored in order to quantify andextrapolate noise levels. Noise and vibrational measurement devices maybe global and monitor a region of a Structure, or they may be inherentlyincorporated into or upon individual equipment of the Structure.

In some examples, models of a Structure (including original models andAs Built models) may include routings of pipes, wires, conduits andother features of a Structure and the installed equipment that havestructure. Together with models of the building structure and theequipment placed in the building the various routed structures may bemarried in a detailed AVM 201.

In another aspect, an AVM 201 may include conflicts between the physicalstructures may be detected and avoided in the design stage at farimproved cost aspects. In some examples, a designer may virtuallyascertain a nature of the conflict and alter a design in virtual spaceto optimize operational aspects. Additionally, in some embodiments, anAs Built model may be generated during and after a Structure is builtfor various purposes. In some examples, a technician may inspect aStructure for conformance of the build to the designed model. In otherexamples, as an As Built Structure is altered to deal with neededchanges, changes will be captured and included in the As Built AVM 201.

In another aspect of the present invention, the AVM 201 may be used togenerate a virtual reality model of a Property, including one or morestructures that may be displayed via user interface that includes animmersion of the user into a virtual setting. Immersion may beaccomplished, for example, via use of a virtual reality headset withvisual input other than a display screen is limited. In someembodiments, a virtual setting may be generated based upon a location ofthe user. For example, GPS coordinates may indicate a Property and auser may wear a headset that immerses the user in a virtual realitysetting. The virtual reality setting may display one or more virtualmodels of structures that may be potentially constructed on theProperty.

Embodiments may include models generated, standard modeling softwaresuch as BIM 360™ field which may support the display of a Structuredesign in a very complete level of detail. Modeling of a Structure inits location or proposed location, or in multiple proposed locations,may be useful from a Total Cost of Ownership perspective, especiallyfrom an evaluation of the nature of a site layout including real estateproperty parcel options and the like.

In some examples, a virtual display observed in the field at the site ofan As Built or proposed build may allow for design changes and designevaluations to be viewed in a space before build is completed. Forexample, a structure may be completed to the extent that walls, floorsand ceilings are in place. A user may utilize a virtual display tounderstand the layout difference for different designs and the designsmay be iterated from designs with the least flexibility to more flexibleyet more complex designs.

In some examples, the design systems may include various types offeatures such as building structure, walls, ducts, utilities, pipes,lighting, and electrical equipment. The design systems are augmentedwith As Built Data and Experiential Data.

The design and modeling systems may be utilized to simulate and projectcost spending profiles and budgeting aspects. The modeling systems maytherefore be useful during the course of an audit, particularly whencomparing actual versus projected spending profiles. The comparison ofvarious spend sequencing may be used to optimize financing costs,maintenance, refurbishing and sequencing. The AVM 201 may be useful toprovide early estimates, and for cost tracking versus projections whichmay be visualized as displays across a virtual display of the building,facilities and equipment.

Energy/Utilities Cost: There may be numerous examples of tradeoffs insources of electric energy to a Structure. For example, a site may bedesigned with various utility supplies for power, with tailored powermanagement systems to balance the capacitance and impedance of theeffective load to minimize electricity cost. In addition, variousalternative forms of electric energy may be assessed and designed.Solar, geothermal and Wind generated electric power may make economicsense under certain conditions and may have time of day and seasonalrelevance. The design of flexible support facilities for theinstallation of initial energy generation capacity with provision forthe addition of additional capacity may be assessed. In some instances,backup power generation may be designed to ensure that a Structure mayrun at some level for a certain period of time. In some cases, this mayallow for continued production, in other examples, backup power may givea Structure the time to idle and shut down capacity in a safer and lessdamaging manner.

In some examples, an energy source for heating, cooling, humidificationand dehumidification equipment may be modeled and managed. In someexamples, a source of energy used may be one or more of electric,natural gas, propane, fuel oil or natural gas. Emergency backup may alsobe modeled and managed. Various choices between electric sources. Solarand fuel based energy consumption may be modeled and controlled based onupon market forecasts. Estimates may be periodically adjusted accordingto world and/or market events.

Enhanced inspection, and guidance capabilities enabled via ongoingelectronic Sensor measurements may facilitate one or more of:maintenance, expansion and optimization of Structure features, operationProperty equipment and maintenance models. Ongoing monitoring via Sensordata collection also increases knowledge of machines and operations, orother useful capacities towards knowing the state of the Structure.

Decisions related to maintenance of equipment and facilities may beimportant decisions that modeling and operational management systemssupport. The various cost elements that may go into modeling mayinclude, for example, one or more variables related to consumables, suchas: a cost of consumables; frequency of replacement 241, quantity ofconsumables 242, life of replaced parts, nature of failures of differentpart types; manpower associated with planned and unplanned maintenanceand expected and actual life of equipment

Inside of a functional Structure, augmented reality functions viewablein an AVM 201 including an AVM may be used to guide operators,surveyors, repair workers, or other individuals, through the Structure.As one non-limiting example, a tablet, mobile device, or other smalldevice with a screen, imaging, and other sensing capabilities may beused in an augmented reality fashion towards this function.

As described above, facing a mobile device towards an area in aStructure and movement of the mobile device in a particular pattern maybe used to ascertain a specific area of the Structure for which AVM 201data should be accessed. A combination of one or more of: image,location, orientation, and other Sensors may also be used to identify tothe mobile device, which wall segment, building aspect, machinery orequipment the device is identifying. A location of mobile device, aheight and an angle of view may also be utilized to determine aspects ofthe structure for which a virtual model is being requested.

In some embodiments, a user may be presented with various layers ofdata, including, for example, one or more of: structural aspects of theStructure, plumbing, electrical, data runs, material specifications orother documentation, including but not limited to: basic identifyinginformation, installation information, service records, safety manuals,process records, expected service schedule, among many otherpossibilities.

A plurality of information may be thus easily accessible inside theStructure, and may be used for a variety of functions, including findinga specific machine to then diagnose and service a problem, regularinspection of equipment, guided tours of the Structure, or many otherfunctions. This information may be conveyed to the individual in aplurality of possible formats, such as lists that show up on the screen,clickable icons that show up next to the equipment in a Virtual Reality(“VR”) camera feed, or many other possibilities. These functions mayalso be accessible in a hands-free VR format with a VR headset, or othersuch device.

As the user is inside a Structure, the user may receive a plurality ofinformation, instructions, etc. while the user is proximate to thevarious aspects of the structures. For example, the user machinesthemselves, seeing them work, hearing the sounds they make, etc. tobetter inspect or service, among other possible functions, theStructure's equipment. With VR systems, similar travel, guidance, orinspection capabilities for a functional Structure may be achievedcompletely remotely from the Structure itself. Additionally, with VRsystems, these capabilities may occur prior, during, or after theconstruction and deployment of a Structure.

A VR system may constitute a headset or lens system with stereoscopicviewing capabilities, a sound conveying means, such as headphones, andvarious forms of user input, such as a handheld controller or footpedals as non-limiting examples. Various forms of imaging, surveying, ormodeling technology may be used to generate virtual models of afunctional Structure. As a non-limiting example, exploring such a modelwith a VR system may be used to examine layout, functioning, or otherparameters of a Structure before its construction. As an alternativenon-limiting example, exploring a model possibly generated by sensingtechnology in real time, or over a period of time prior to viewing witha VR system, may allow for inspection or demonstration capabilities in alocation entirely remotely from the actual Structure itself. This mayinclude both imagery and sounds captured within the Structure.

Collection of data may additionally include actual service lifeexperienced and performance of equipment used in an AVM which therebyenables enhanced modeling of a life expectancy of equipment included inan Augmented Virtual Model 100 and an As Built structure. VariousSensors may gather relevant data related to one or more of: use ofmachinery and equipment, performance of machinery items of equipment andan ambient environment inside or proximate to machinery and equipment.In addition, an unstructured query relating to the functioning or lifeexpectancy of equipment may be generated by a processor to access andinterpret data, thereby deriving relevant input to a decision makerbased upon analysis of the data.

Various examples of data to be acquired, relating to life expectancy ofequipment, may include, but is not limited to, hours of operation,conditions of operation (whether and how long the equipment may berunning under capacity, at rated capacity, or over capacity), or manyenvironmental conditions for operation; environmental conditions mayinclude the ambient temperature (or the difference in ambienttemperature from an ideal or other measured value), ambient humidity (orthe difference in ambient humidity from an ideal or other measuredvalue), ambient air particulate content (or a comparison of the currentair particulate level to a filter change schedule), presence orconcentration of ambient gasses (if relevant) such as carbon dioxide, orother gas, a number of times of ingress or egress into the Structurewhich may change ambient conditions or other trackable data.

Identification of Equipment

Identification capabilities may be facilitated or improved for one ormore of: structural aspects, machinery, equipment and utility supportwithin the Structure. This identification may take many forms throughvarious means of query and communication, and may be facilitated throughvarious hardware and/or software means.

Non-limiting examples may include image based identification; a devicewith some imaging means, including but not limited to a mobile devicecamera, tablet device camera, computer camera, security camera, or ARheadset camera may image the equipment to be identified. Imagerecognition software may be used to identify the visualized equipment byits identifying features. Machine learning may be used to train systemsusing this software to identify specific features of the equipment inquestion. Other types of visual identifiers including but not limited toQR codes, may be used to visually identify equipment.

An additional non-limiting example may include location basedidentification; a device with some location means, including but notlimited to GPS, internal dead-reckoning, or other means, may be used todetermine a location within a Structure. Identifying information forequipment at or near the measured location may be accessed forassessment, based on its proximity to the location based signal.

An additional non-limiting example may also include direction basedidentification; with a fixed location, or in tandem with a locationmeans, a device may have capabilities to deduce orientation basedinformation of the device. This orientation information may be used todeduce a direction that the device is pointing in. This direction basedinformation may be used to indicate that the device is pointing to aspecific piece of equipment that may be identified.

An additional non-limiting example may also include As Built sensor andsensor generated experiential data based identification; identifyinginformation for various equipment may be stored and accessed within adatabase storing this information. This information may be accessed byvarious means by a user with certain qualification to that information.

An additional non-limiting example may include tag based identification;identifying information for various equipment may be accessed throughproximity to many non-limiting examples of tagging capabilities, such asmagnetic tags, bar code tags, or others. These tags may contain theinformation in question, or may reference the location of pertinentinformation to the owner, in order to convey this information to theowner.

An additional non-limiting example, data aggregation may include sensorsgenerating data that is associated with an IoT (Internet of Things)based identification. Various IoT devices (or Sensors) may include adigital storage, processor and transmitter for storing and conveyingidentifying information. Upon request, an IoT device may relayidentifying information of itself to a human with a communicatingdevice, or to its neighbors. It may also possibly convey informationreceived from and/or sent to other internet connected devices as well.

Data aggregated and stored for reference in calculation of Cost ofUpkeep considered in a TOC and may include data related to some or allof:

-   -   Documented items covered;    -   Long term warranty for Structure/building ownership;    -   Items included in purchase price;    -   financed amounts;    -   Tax implications;    -   Capital value;    -   Ability to expand Structure and/or structural features such as        baths or kitchens;    -   Lateral dimensions;    -   Vertical dimensions;    -   Building support systems;    -   Utilities;    -   Electric;    -   Water;    -   Discharge;    -   Aggregate Data;    -   Same Structure;    -   Multiple similar facilities;    -   Disparate Structure types;    -   Same geographic area;    -   Disparate geographic areas;    -   Locating Machine s and Equipment;    -   GPS (may be used in combination with other location        technologies;    -   Near field communication with reference point emitter in        Structure;    -   WiFi;    -   RFID;    -   Reflector tags;    -   “Visual” recognition identifiers, i.e. hash, barcode; and    -   Directional—accelerometers in combination with visual        recognition identifiers.

As per the above listing, functionality may therefore include modeledand tracked Performance of a Structure and equipment contained withinthe Structure, including consumables 233 used and timing of receipt andprocessing of consumables; modeled and actual maintenance 232, includingquality of maintenance performed; equipment Performance includingyields; Consumables 233 tracking may include a frequency of replacementand quantity of replaced consumables; Utilities 234 tracking may includeprojected and actually units of energy consumed.

3D Scanning & Model Development

In one aspect of the present invention data related to the position andidentity of substantial elements of a Structure are first designed andthen recorded in their actual placement and installation. This mayinclude locations of building features, such as beams, walls, electricaljunctions, plumbing and etc. as the structure is designed andconstructed. As part of the Structure model, laser scanning may beperformed on site at various disparate times during construction. Aninitial scan may provide general information relating to the location ofthe structure in relationship to elements on the property such asroadways, utilizes such as electricity, water, gas and sewer to identifynon-limiting examples.

Additional events for scanning may occur during the construction processin order to capture accurate, three-dimensional (3D) “as-built” pointcloud information. Point cloud may include an array of points determinedfrom image capture and/or laser scanning or other data collectiontechnique of As Built features. In some examples, captured data may beconverted into a 3D model, and saved within a cloud-based data platform.

In some examples other methods of capturing spatially accurateinformation may include the use of drones and optical scanningtechniques which may include high resolution imagery obtained frommultiple viewpoints. Scanning may be performed with light based methodssuch as a CCD camera. Other methods may include infrared, ultraviolet,acoustic, and magnetic and electric field mapping techniques may beutilized.

Structure related information may include physical features generallyassociated with an exterior of a structure such as geo-location,elevation, surrounding trees and large landscaping features, undergroundutility locations (such as power, water, sewer, sprinkler system, andmany other possible underground utility features), paving, and pool orpatio areas. Structure related information may also include featuresgenerally related to a structure such as underground plumbing locations,stud locations, electrical conduit and wiring, vertical plumbing piping,and HVAC systems or other duct work. The acquisition of the data mayallow the model system to accurately locate these interior and exteriorfeatures. Acquisition of As Built data during different points of theconstruction completion allows measurements to be taken prior to aspectsinvolved in a measurement process being concealed by concrete, sheetrockor other various building materials.

Data is acquired that is descriptive of actual physical features as thefeatures are built and converted into a 3D model which may be referredto as the “As Built” model. The As Built model will include “keycomponents” of the structure and be provided with a level of artificialintelligence that fully describes the key component. In someembodiments, the As Built model may be compared to a design model. Insome implementations “intelligent parameters” are associated with keycomponents within the 3D model. For example, key components andassociated information may further be associated with intelligentparameters. Intelligent parameters for the key components may includethe manufacturer, model number, features, options, operationalparameters, whether or not an option is installed (and if so, itsfeatures and dimensions), any hardware associated with the key component(and its manufacturer and serial number), an owner's manual and servicecontract information, as non-limiting examples. Intelligent parametersassociated with a functional key component such as, HVAC Equipment, mayinclude the manufacturer, model number, capacity, efficiency rating,serial number, warranty start date, motor size, SEER rating, an owner'smanual associated with the equipment, and service contract information.

Key components of the structure may have an identification device suchas a two or three dimensional graphical code (such as a QR code label) aRadio Frequency Identification Chip (RFID) attached that is accessibleto a user, such as a structure owner, structure builder or servicetechnician. When scanned with an apparatus capable of reading the code,a user interface on a display of various types, such as a tablet, mayuse the associated identification, such as a QR code, to provide directaccess to related information. In some examples, the display may showtextual or tabular representations of related data.

In other examples, graphical data such as images, drawings, and the likemay be displayed. In still further examples, both graphical and textualdisplays may be associated with the code. Although a QR code may providean example, other identification technologies such as radio frequencyID, Internet of things (IoT) communication protocols with associatedstored information, and other devices that can receive a signal andrespond with stored information may be used. As well, numerous othertypes of graphical codes in addition to QR code may be read by a deviceand provide a connection between a key component, machinery, locationand other identified aspect and associated data. In some examples, animage based code may be displayed using paints or pigments which are notvisible to the human eye, such as in a non-limiting example ultravioletpigments. In some other examples, a paint or pigment may not be visibleuntil it is made to emit visible light by irradiating it with aparticular band of electromagnetic radiation, such as, for example,ultraviolet light.

In some examples, key components may include doors, windows, masonry,roofing materials, insulation, HVAC equipment and machinery.

An automated Design and Monitoring (“RDM”) system may support dynamicupdating of tracked aspects. For example, as a structure owner acquiresnew or additional key components, such as machinery, HVAC, plumbingadditions, key components may be added into the As Built model and thekey components may be tracked as a part of the model. Other aspects maybe dynamically updated such as when additions are made to the buildingstructure or rebuilding of internal structure is made as non-limitingexamples.

Since the As Built model includes information in a database and dynamicmodel functionality exists that commences as a building structure isbeing constructed, the model may assume new support aspects to theconstruction process itself. For example, a benefit from the definitionand utilization of many components within a Structure utilizing thesystem herein includes the ability to pre-cut and/or pre-fabricate studsand framing, roofing cuts, masonry, under-slab plumbing, HVAC ductwork,electrical, and other such components. The dimensions of these variouscomponents may be dynamically updated based on an original model thatmay be compared to actual fabricated structure as realized on a buildingsite. In some examples a structure builder may use a display interfaceassociated with the system and model to display a comparison of anoriginal set of building plans to a current structure at a point in timewhich may allow the builder to authorize any structural changes orvariances to design and thereafter allow the description of followingcomponents to be dynamically adjusted as appropriate. The system may beof further utility to support various inspections that may occur duringa building project which may associate detected variances with designexpert review and approval. An inspector may be able to utilize thesystem as allowed on site or operate a window into the system from aremote location such as his office.

As the system is utilized during construction, orders for customizedcomponents may be placed. These customized components may be labeled anddelivered to site, in an appropriate sequence, for assembly bycarpenters. This may contribute to a minimization of waste at theworksite, as well as provide a work product that is entirely consistentwith a pre-determined model which may have approved changes that aretracked. The result may improve the quality of the work product, andmake it easier to generate the measured point-cloud 3D model.

Performance Tracking

In another aspect, the AVM system can autonomously and/or interactivelyobtain, store and process data that is provided to it by components ofthe Structure as the structure is built, installed or additions are madeto the structure. The generation, modeling, capture, use, and retentionof data relating to Performances in specific equipment or in some casesaspects relating to the design of a facility, may be monitored by thesystem.

In some examples, Operational Performance may be assessed by processingsampled data with algorithms of various kinds. Feedback of the status ofoperation and of the structure as a whole or in part, as assessed byalgorithmic analysis may be made to a structure owner or a structurebuilder. In addition, a variety of data points gathered via appropriateSensors, visual and sound data may be recorded and stored and correlatedto 3D models of the facility. Experiential Sensor readings may include,by way of non-limiting example: temperature, power usage, utilitiesused, consumables, product throughput, equipment settings, and equipmentPerformance measurement, visual and audible data. Techniques to recorddata points may involve the use of one or more of: electronic Sensors,electro-mechanical Sensors, CCD capture devices, automated inspectionequipment, video camera arrays and audio microphones and arrays of audiomicrophones for the capture and processing of data that may be used togenerate visualizations of actual conditions, either on site or at aremote location. In addition, data may be collected, retained, analyzed,and referenced to project facility Performance.

In some examples, data may also be combined with manufacturer equipmentspecifications and historical data to model expectations related toactual operation of the structure and property aspects.

Virtual Maintenance Support

A 3D model of structure, such as a structure, which may be integratedwith information related to the key components and laser scannedlocation information, may be made available to the structureowner/structure builder through a computer, an iPad or tablet, or smartdevice. The resulting system may be useful to support virtualmaintenance support.

The three dimensional model may support enhancement to the twodimensional views that are typical of paper based drawings. Althoughthree dimensional renderings are within the scope of informationdelivered in paper format, a three dimensional electronic model mayrender dynamic views from a three dimensional perspective. In someexamples, the viewing may performed with viewing apparatus that allowsfor a virtual reality viewing.

In some examples, a viewing apparatus, such as a tablet or a virtualreality headset, may include orienting features that allow a user suchas a structure owner, structure builder, inspector, engineer, designeror the like to view aspects of a model based upon a location, adirection, a height and an angle of view. A current view may besupplemented with various other information relating to featurespresented in the view. In some examples, the interface may be accessiblethrough a virtual reality headset, computer, or mobile device (such asan iPad, tablet, or phone), as non-limiting examples. Utilizing a deviceequipped with an accelerometer, such as a virtual reality headset ormobile device, as non-limiting examples, a viewable section of the modelmay be displayed through the viewing medium (whether on a screen, orthrough a viewing lens), where the viewer's perspective changes as theaccelerometer equipped device moves, allowing them to change their viewof the model. The viewer's Vantage Point may also be adjusted, through acertain user input method, or by physical movement of the user, asnon-limiting examples.

The presented view may be supplemented with “hidden information”, whichmay include for example, depictions of features that were scanned beforewalls were installed including pipes, conduits, ductwork and the like.Locations of beams, headers, studs and building structure may bedepicted. In some examples, depiction in a view may include asuperposition of an engineering drawing with a designed location, inother examples images of an actual structure may be superimposed uponthe image based upon As Built scans or other recordations.

In a dynamic sense, display may be used to support viewing ofhypothetical conditions such as rerouted utilities, and rebuild wallsand other such structure. In some examples, graphical or text based datamay be superimposed over an image and be used to indicatespecifications, Performance aspects, or other information not related tolocation, shape and size of features in the image.

As presented above, an image may allow for a user to “see through walls”as the augmented reality viewing device simulates a section of a modelassociated with a space displayed via the virtual reality viewingdevice. The viewer's perspective may change as an accelerometer in thevirtual reality viewing device moves. A user may also change a view ofthe AVM, to include different layers of data available in the AVM. Theviewer's Vantage Point may also be adjusted by moving about a physicalspace that is represented by the model. To achieve this, it may bepossible to incorporate positioning hardware directly into a buildingrepresented by the virtual model. The positioning hardware may interfacewith an augmented reality device for positioning data to accuratelydetermine the viewing device's orientation and location with millimeterprecision. The positioning hardware may include, for example a radiotransmitter associated with a reference position and height. Altitude isdifferentiated from height unless specifically referenced since therelative height is typically more important.

Accordingly, a user may access the AVM on site and hold up a smartdevice, such as an iPad or other tablet, and use the smart device togenerate a view inside a wall in front of which the smart device ispositioned, based upon the AVM and the location, height and direction ofthe smart device position.

In some examples, through the use of an augmented reality device, it mayalso be possible to view data, such as user manuals, etc. of associateddevices in the view of a user, simply by looking at them in the viewinginterface. In other examples, there may be interactive means to selectwhat information is presented on the view.

Various electronic based devices implementing of the present inventionmay also be viewed in a virtual reality environment withoutaccelerometer such as a laptop or personal computer. A viewable sectionof a model may be displayed on a Graphical User Interface (GUI) and theviewer's Vantage Point may be adjusted, through a user input device.

The ability to track machinery and other components of a system andstore the components associated information, such as, for example usermanuals and product specifications and part numbers, may allow for muchmore efficient use and maintenance of the components included within astructure. As well, the system model may also maintain structure ownermanuals and warranties and eliminate the need for storage and trackingof hard copy manuals.

In a non-limiting example, if a structure owner/structure builderdesires information related to an machinery, it may be found bypositioning a device with a location determining the device within it inproximity to the machinery and accessing the parallel model in theVirtual Structure such as by clicking on the machinery in the VirtualStructure model or by scanning the Code label attached to machinery. Insome examples, an internet of things equipped machine may have theability to pair with a user's viewing screen and allow the system modelto look up and display various information. Thus, the user may haveaccess to various intelligent parameters associated with that machinerysuch as service records, a manual, service contract information,warranty information, consumables recommended for use such asdetergents, installation related information, power hooked up and thelike.

In some examples, an AVM system may include interfaces of various kindsto components of the system. Sensors and other operational parameterdetection apparatus may provide a routine feedback of information to themodel system. Therefore, by processing the data-stream with variousalgorithms autonomous characterization of operating condition may bemade. Therefore, the AVM system may provide a user with alerts whenanomalies in system Performance are recognized. In some examples,standard structure maintenance requirements may be sensed or trackedbased on usage and/or time and either notification or in some casesscheduling of a service call may be made. In some examples, the alertmay be sent via text, email, or both. The structure user may,accordingly, log back into the Virtual Structure to indicate completionof a maintenance task; or as appropriate a vendor of such service ormaintenance may indicate a nature and completion of work performed.

By detecting operational status, a Virtual Structure may take additionalautonomous steps to support optimal operation of a system. A VirtualStructure may take steps to order and facilitate shipping of anticipatedparts needed for a scheduled maintenance ahead of a scheduled date for amaintenance event (for example, shipping a filter ahead of time so thefilter arrives prior to the date it is scheduled to be changed). Inanother example, a Virtual Structure may recall notes from an OriginalEquipment Manufacturer (OEM) that could be communicated to a userthrough the Virtual Structure. In still further examples, a VirtualStructure may support a user involved in a real estate transaction byquantifying service records and Performance of a real property.

In still another aspect the AVM may establish a standard maintenance andwarranty program based on manufacturers published data and the abilityto advise structure owners of upcoming needs and/or requirements. Inother examples, the model system may facilitate allowing for structurebuilders, rental companies, or maintenance companies to consolidateinformation for volume discounts on parts or maintenance items. Themodel system may also facilitate minimizing unnecessary time expenditurefor structure builders hoping to minimize needless service calls forwarranty issues, and allowing structure builders and rental companiesattempting to sell a structure or a rental to demonstrate that care hasbeen taken to maintain a structure.

Benefits derived from monitoring and tracking maintenance with a VirtualStructure may include positively reassuring and educating lenders and/orlien holders that their investment is being properly cared for. Inaddition, insurance companies may use access to a Virtual Structure toprovide factual support that their risk is properly managed. In someexamples, a data record in a Virtual Structure model system and how anowner has cared for its facility may be used by insurance companies orlenders to ensure that good care is being taken. Maintenance recordsdemonstrating defined criteria may allow insurance companies to offer astructure owner policy discount, such as, for example, installation ofan alarm system. Additionally, access to a Virtual Structure may allowmunicipalities and utilities to use the info for accurate metering ofutility usage without having to manually check; and peaks in utilitydemand may be more accurately anticipated.

In some examples, Virtual Structure may also be used to assist withstructure improvement projects of various types. In some examples, thestructure improvement projects may include support for building largeradditions and modifications, implementing landscaping projects. Smallerprojects may also be assisted, including in a non-limiting example sucha project as hanging a picture, which may be made safer and easier withthe 3D “as-built” point cloud information. Hidden water piping,electrical conduits, wiring, and the like may be located, or virtually“uncovered”, based on the model database.

Optimization of Facilities

During construction of a structure corresponding to a Virtual Structure,discrete features of the As Built structure may be identified via anidentification device such as an IoT device or a QR code label. The IDdevice may be integrated to the feature or added during the build scope.Performance monitors may also be simultaneously installed to allowmonitoring of Key Performance Indicators (KPIs) for selected features.In an example, an HVAC system may be added to a facility duringconstruction and a simultaneously a Performance monitor may be added tothe HVAC system. The Performance monitor may be used to monitor variousKPIs for an HVAC system. These KPIs may include outdoor air temperature,discharge air temperature, discharge air volume, electrical current, andthe like. Similar monitoring capabilities may be installed to allmachinery and utilities systems in a facility. The combination of thesenumerous system monitors may allow for a fuller picture of theefficiency of operations of various systems.

Use of the Virtual Structure, which may include data values contributedfrom communication of data from the various monitoring systems, mayallow owners to receive periodic reports, such as in a non-limitingsense monthly emails which may show their current total energyconsumption as well as a breakdown of what key components arecontributing to the current total energy consumption.

The systems presented herein may be used by owners and facility managersto make decisions that may improve the cost effectiveness of the system.An additional service for Owners may allow the structure owner to tapinto energy saving options as their structure ages. As an example, if amore efficient HVAC system comes on the market, which may includeperhaps a new technology node, the user may receive a “Savings Alert”.Such an alert may provide an estimated energy savings of the recommendedmodification along with an estimate of the cost of the new system. Theseestimates may be used to generate a report to the owner of an estimatedassociated return-on-investment or estimated payback period should thestructure owner elect to replace their HVAC system.

In some examples, a AVM of a Virtual Structure may set a threshold valuefor the required ROI above which they may be interested in receivingsuch an alert with that ROI is achieved. This information will be basedon data derived from actual operating conditions and actual historicalusage as well as current industry information. Predictive maintenanceand energy savings to key systems via Smart Structure Total Cost ofOwnership (“TCO”) branded Sensors.

Aggregating Data from Multiple Residences

With the ability to collect and utilize relevant structure informationwith the model system, the aggregation of data and efficiency experiencefrom numerous systems may allow for analysis of optimization schemes forvarious devices, machinery and other structure components that includesreal installed location experience. Analysis from the aggregated datamay be used to provide feedback to equipment manufacturers, buildingmaterials fabricators and such suppliers.

In some examples, business models may include providing anonymous andaggregated data to original equipment manufacturers as a service modelto give the OEMS an ability to utilize more data to monitor and improvetheir products. In some examples, OEM advertising may be afforded accessthrough the model system. Manufacturers may have an additional sidebenefit motivating the use of this data related to improving theirequipment cost effectives and reliability in order to minimize warrantycost. Such optimized Performance may also provide benefits to bothstructure owners and builders to support their ability to track actualwarranty information, power cost, and overall Performance of astructure.

Methods and Apparatus

Referring to FIGS. 3A-3F, an illustration of the collection of data byscanning a facility during its construction is provided. In FIG. 3A, adepiction of a site for building a facility structure is illustrated.The depiction may represent an image that may be seen from above thesite. Indications of property boundaries such as corners 301 andproperty boarders 302 are represented and may be determined based onsite scanning with property markings from site surveys or may be enteredbased on global coordinates for the property lines. An excavatedlocation 303 may be marked out. Roadways, parking and/or loading areas304 may be located. Buried utilities such as buried telephone 305,buried electric 306, buried water and sewer 307 are located in the modelas illustrated. In some examples, such other site service as a buriedsprinkler system 308 may also be located.

Referring to FIG. 3B the excavated location 303 may be scanned or imagedto determine the location of foundation elements. In some non-limitingexamples, a foundational footing 321 along with buried utilities 322 isillustrated. The buried utilities may include such utilities as electriclines, water supply whether from a utility or a well on location, seweror septic system lines, telecommunications lines such as telephone,cable and internet. Other footing elements 323 may be located atstructural requiring locations as they are built. In some examples ascanning system may provide the locational orientation relative to siteorientation markings. In other examples, aerial imagery such as may beobtained with a drone may be used to convert features to accuratelocation imagery.

Referring to FIG. 3C a wall 331 of the Structure in the process of buildis illustrated. The structure may be scanned by a scanning element 330.In some examples, a laser three dimensional scanner may be used. Thewall may have supporting features like top plates 333, headers 336,studs 332, as well as internal items such as pipes 334, electricalconduits and wires 335. There may be numerous other types of featureswithin walls that may be scanned as they occur such as air ducts, datacables, video cables, telephone cables, and the like.

Referring to FIG. 3D the wall may be completed with structure componentsbehind wall facing 340 may no longer be visible. Electrical outlets 341and door structures 342 may be scanned by a scanning element 330.

Referring to FIG. 3E internal components such as machinery may beinstalled. As a non-limiting example, a machine 350 may be installed andthe resulting three dimensional profiles may be scanned by a scanningelement 330. In some examples, an operational monitor 351 may beattached to the machinery. In some examples, an operational monitor maybe part of the machinery. The operational monitor may have the abilityto communicate 352 data to various receivers that may be connected tothe model system of the residence. In some examples, key structuralcomponents, such as doors, may have identifying devices such as a QRlabel 353. The label may be visible or painted into the structure withnon-visible paint. The identifying devices may provide informationrelated to the device itself and warrantees of the device asnon-limiting examples.

The model may include the various structure elements hidden and visibleand may be used to create output to a display system of a user.Referring to FIG. 3F an example display is illustrated. The variousnon-visible layers may be shown by rendering the covering layers with atransparency. Thus the display shows the machine profile 350 as well asthe internal features that may be concealed like pipes 334, electricalconduits with wires 335, and headers 336 as examples.

Referring to FIG. 3G, an illustration of feedback of the model system isillustrated. A wall that has been scanned with an HVAC unit 360 mayinclude a Performance Monitor 351 which may communication variousinformation wirelessly 352. The communication may be received at anantenna 370 of a router 371 within the facility. The facility may beinterconnected through the internet 372 to a web located server 373which processes the communication. The web located server 373 also caninclude the various model data about the facility and it can providecomposite displays that can summarize the structure as well as theoperational Performance of the HVAC unit 360. It may aggregate thevarious data into textual and graphic reports. In some examples it maycommunicate these reports back through internet connections. In otherexamples, wireless Smart Device communications may be sent to cellulartowers 374 which may transmit 375 to a Smart Device 376 of a userassociated with the facility.

Referring to FIG. 3H an illustration of a virtual reality display inconcert with the present invention is illustrated. A machinery 350 ofthe facility may communicate information to the model server. A user 380may receive may an integrated communication from the server. Theresulting communication may be provided to a virtual reality headset381. The virtual reality headset may provide a display 382 to the userthat provides a three-dimensional view of the physical data as well assimulated imagery that may allow views through objects to hiddenelements behind the object. As well, a heads up type display ofinformation about an object may be superimposed.

Referring now to FIG. 4A, method steps that may be implemented in someembodiments of the present invention are illustrated. At method step401, Deployment aspects may be specified for a Structure andincorporated into a virtual model, such as an AVM discussed above.Deployment aspects may include for example, a purpose for an As Builtstructure that is built based of the AVM. The purpose may include, byway of non-limiting example, one or more of: manufacturing, processing,data processing, health care, research, assembly, shipping andreceiving, prototyping and the like.

Deployment aspects may also include a level of use, such continual,shift schedule or periodic. A climate in which the structure will beplaced may also be considered in the Deployment aspects. Climate mayinclude one or more of: four seasons; primarily winter; tropical,desert; exposed to salt air; and other environmental factors.

At method step 402, a virtual model, such as an AVM is digitally createdaccording to the Deployment aspects of the model. The AVM may includeimprovements to a real estate parcel and a structure that will be placedon the real estate parcel, as well as where a structure may be locatedupon the parcel.

At method step 403, Performance aspects of machinery that may beincluded in the AVM may be digitally modeled and may include a level ofuse of the machinery and an expected satisfaction of the machinery asdeployed according to the Deployment aspects. Maintenance expectations,including a number of repair calls and a preventive maintenance schedulemay also be modeled and associated costs.

At method step 404, Performance aspects of equipment that may beincluded in the AVM may be digitally modeled and may include a level ofuse of the equipment and an expected satisfaction of the machinery asdeployed according to the Deployment aspects. Maintenance expectations,including a number of repair calls and a preventive maintenance schedulemay also be modeled and associated costs.

At method step 405, As Built aspects of a structure are recorded asdiscussed herein, preferably recordation of As Built aspects begins asconstruction begins and continues throughout the existence of thestructure.

At method step 406, the physical structure may be identified via alocation. A physical location may include, for example, CartesianCoordinates, such as Latitude and Longitude coordinates, GPScoordinates, or other verifiable set of location parameters. Inaddition, more exact location specifications may include surveydesignations.

At method step 407, a position within or proximate to the Structure maybe determined via positioning identifiers. The position within orproximate to the Structure may be determined.

At method step 408, an AVM may be identified and accessed via thephysical location. Once an appropriate AVM is accessed, a particularportion of the AVM may be presented via a GUI based upon the positionwithin the Structure (or proximate to the Structure) and a direction,height and angle of view. The position may be determined relative tolocation identifiers. Height may be determined via electronic devices,such as a smart device, or via triangulation referencing the locationidentifiers (locations identifiers are discussed more fully above andbelow).

At method step 409 an update may be made to a physical Structure and atmethod step 410, the update to the physical structure may be recordedand reflected in the AVM.

Referring to FIG. 4B, a method flow diagram for monitoring andmaintenance is illustrated. At 411 a user may obtain a scanning deviceor devices that may scan a building site. At 412, the user or a serviceof the user may mark property boundaries of the site. At 413, work onthe site may continue with the excavation of a building base and thelaying down of utilities and other buried services. At 414, the scanningdevice is used to scan the location of the various aspects of thebuilding site. At 415, work may continue with the laying of footings andfoundations and other such foundational building activities. At 416,scanning of the footings and foundations may be accomplished. At 417, astructure may be framed and features such as pipe conduit, electricalwiring communications wiring and the like may be added. At 418, thebuilding site may again be scanned to locate the various elements. Theframing of the residence may commence along with running of pipe,wiring, conduits, ducts and various other items that are located withinwall structures. Before coverings are placed on walls, the framedstructure may be scanned at 418. Thereafter, the framed structure may beenclosed with walls 419.

Referring to FIG. 4C a method flow diagram for structure monitoring andmaintenance is illustrated. In this flow diagram, a Structure mayalready be built and may have various data layers already located in themodel system. At 421, machinery may be added to the Structure. At 422,an ID tag, or a QR tag, or and RFID tag or an internet of things devicemay be associated with the machinery and may be programmed into themodel system. At 423, the model system may be interfaced to themachinery ID and into the Structure model. At 424, a scanning step maybe used to input three dimensional structure data at the installedlocation into the model system. At 425, an operational monitor functionof the device may be added or activated. At 426, operational data may betransferred from the operational monitor to the server with theStructure model.

At 427, algorithms running on a server of the model system may determinean operational improvement opportunity based on calculations performedon the data from the operational monitor. At 428 a user may query theoperational data of the machinery for information on its warranty. At429, the model system may initiate an order for a service part and mayschedule a service visit to make a repair based upon analysis of theoperational data. The various steps outlined in the processing flow maybe performed in different orders. In some examples additional steps maybe performed. In some examples, some steps may not be performed.

In some embodiments, the present invention includes a method of trackingattainment of a stated Performance Level relating to a Structure,including: a) determining a geographic position of a Structure via aglobal positioning system device in a smart device proximate to theStructure; b) identifying a digital model of the Structure based uponthe geographic position of the Structure, the digital model comprisingvirtual representation of structural components included in theStructure; c) referencing multiple positioning reference devices withinthe Structure; d) measuring a distance to at least three of the multiplepositioning reference devices from a point of measurement; e)calculating a position within the Structure, the calculation based upona relative distance of the at least three positioning reference devicesto the point of measurement and a triangulation calculation; f)calculating an elevation of the point of measurement; g) measuring afirst state within the Structure with a sensor; h) specifying a locationof the first state within the Structure via reference to the position ofthe point of measurement and the elevation of the point of measurement;i) recording a first time designation for the step of measuring a firststate within the Structure with a sensor; and i) correlating the firststate within the Structure and the first time designation attainment ofthe stated Performance Level.

The geographic position may be calculated with a GPS reading from withinthe Structure. Measuring a distance to the at least three of thepositioning reference devices may include, one or more of: relativesignal strength received from wireless transmissions emanating from theat least three positioning reference devices; time of arrival of radiosignals of wireless transmissions emanating from the at least threepositioning reference devices measuring a distance to the at least threepositioning reference devices comprises time difference of arrival ofradio signals of wireless transmissions emanating from the at leastthree reference positioning devices.

The above steps may be repeated for at least a second state and a secondtime designation, and in preferred embodiments multiple more states andtime designations.

A state may include, for example, one or more of: a vibration measuredwith an accelerometer; a temperature of at least a portion of thestructure; an electrical current measurement to equipment installed inthe Structure, a number of cycles of operation of equipment installed inthe Structure; a number of cycles of operation of an machinery installedin the Structure; an electrical current measurement to an machineryinstalled in the Structure; a vibration associated with movement of anoccupant of the Structure.

A vibration pattern may be associated with a specific occupant andtracking the movement of the specific occupant through the structure maybe based upon measured vibration patterns. Similarly, a vibrationpattern may be associated with a particular activity of a specificoccupant and the activity of the specific occupant may be tracked withinthe structure based upon measured vibration patterns.

A Performance Level may include one or more of: operating the Structurefor a term of years within a threshold use of energy; operating theStructure for a term of years within a threshold number of repairs; andoperating the Structure for a term of years within a threshold budgetarycost.

FIG. 5 illustrates location and positioning identifiers 501-504 that maybe deployed in a Structure according to some embodiments of the presentinvention to determine a user position 500 within or proximate to theStructure 505. Positioning identifiers may include a device that isfixed in a certain location and may be used to determine via calculationa position of a user with a tablet, smart phone or other network accessdevice able to recognize the position identifiers. The positionidentifiers 501-504 may include devices, such as, for example, a radiotransmitter, a light beacon, or an image recognizable device. A radiotransmitter may include a router or other WiFi device. In someembodiments, a position identifier may include a WiFi router thatadditionally provides access to a distributed network, such as theInternet. Cartesian Coordinates, such as a GPS position 506, may beutilized to locate and identify the Structure 505.

A precise location may be determined via triangulation based upon ameasured distance from three 501-503 or more position identifiers501-504. For example a radio transmission or light signal may bemeasured and compared from the three reference position identifiers501-503. Other embodiments may include a device recognizable via imageanalysis and a camera or other Image Capture Device, such as a CCDdevice, may capture an image of three or more position identifiers501-504. Image analysis may recognize the identification of each ofthree or more of the position identifiers 501-504 and a size ratio ofthe respective image captured position identifiers 501-504 may beutilized to calculate a precise position. Similarly, a heightdesignation may be made via triangulation using the position identifiersas reference to a known height or a reference height.

Referring now to FIG. 6 an automated controller is illustrated that maybe used to implement various aspects of the present invention, invarious embodiments, and for various aspects of the present invention,controller 600 may be included in one or more of: a wireless tablet orhandheld device, a server, a rack mounted processor unit. The controllermay be included in one or more of the apparatus described above, such asa Server, and a Network Access Device. The controller 600 includes aprocessor unit 620, such as one or more semiconductor based processors,coupled to a communication device 610 configured to communicate via acommunication network (not shown in FIG. 6). The communication device610 may be used to communicate, for example, with one or more onlinedevices, such as a personal computer, laptop, or a handheld device.

The processor 620 is also in communication with a storage device 630.The storage device 630 may comprise any appropriate information storagedevice, including combinations of magnetic storage devices (e.g.,magnetic tape and hard disk drives), optical storage devices, and/orsemiconductor memory devices such as Random Access Memory (RAM) devicesand Read Only Memory (ROM) devices.

The storage device 630 can store a software program 640 with executablelogic for controlling the processor 620. The processor 620 performsinstructions of the software program 640, and thereby operates inaccordance with the present invention. The processor 620 may also causethe communication device 610 to transmit information, including, in someinstances, control commands to operate apparatus to implement theprocesses described above. The storage device 630 can additionally storerelated data in a database 650 and database 660, as needed.

Referring now to FIG. 7, a block diagram of an exemplary mobile device702. The mobile device 702 comprises an optical capture device 708 tocapture an image and convert it to machine-compatible data, and anoptical path 706, typically a lens, an aperture or an image conduit toconvey the image from the rendered document to the optical capturedevice 708. The optical capture device 708 may incorporate aCharge-Coupled Device (CCD), a Complementary Metal Oxide Semiconductor(CMOS) imaging device, or an optical Sensor 724 of another type.

A microphone 710 and associated circuitry may convert the sound of theenvironment, including spoken words, into machine-compatible signals.Input facilities may exist in the form of buttons, scroll wheels, orother tactile Sensors such as touch-pads. In some embodiments, inputfacilities may include a touchscreen display.

Visual feedback to the user is possible through a visual display,touchscreen display, or indicator lights. Audible feedback 734 may comefrom a loudspeaker or other audio transducer. Tactile feedback may comefrom a vibrate module 736.

A motion Sensor 738 and associated circuitry convert the motion of themobile device 702 into machine-compatible signals. The motion Sensor 738may comprise an accelerometer that may be used to sense measurablephysical acceleration, orientation, vibration, and other movements. Insome embodiments, motion Sensor 738 may include a gyroscope or otherdevice to sense different motions.

A location Sensor 740 and associated circuitry may be used to determinethe location of the device. The location Sensor 740 may detect GlobalPosition System (GPS) radio signals from satellites or may also useassisted GPS where the mobile device may use a cellular network todecrease the time necessary to determine location. In some embodiments,the location Sensor 740 may use radio waves to determine the distancefrom known radio sources such as cellular towers to determine thelocation of the mobile device 702. In some embodiments these radiosignals may be used in addition to GPS.

The mobile device 702 comprises logic 726 to interact with the variousother components, possibly processing the received signals intodifferent formats and/or interpretations. Logic 726 may be operable toread and write data and program instructions stored in associatedstorage or memory 730 such as RAM, ROM, flash, or other suitable memory.It may read a time signal from the clock unit 728. In some embodiments,the mobile device 702 may have an on-board power supply 732. In otherembodiments, the mobile device 702 may be powered from a tetheredconnection to another device, such as a Universal Serial Bus (USB)connection.

The mobile device 702 also includes a network interface 716 tocommunicate data to a network and/or an associated computing device.Network interface 716 may provide two-way data communication. Forexample, network interface 716 may operate according to the internetprotocol. As another example, network interface 716 may be a local areanetwork (LAN) card allowing a data communication connection to acompatible LAN. As another example, network interface 716 may be acellular antenna and associated circuitry which may allow the mobiledevice to communicate over standard wireless data communicationnetworks. In some implementations, network interface 716 may include aUniversal Serial Bus (USB) to supply power or transmit data. In someembodiments other wireless links may also be implemented.

As an example of one use of mobile device 702, a reader may scan somecoded information from a location marker in a facility with the mobiledevice 702. The coded information may include for example a hash code,bar code, RFID or other data storage device. In some embodiments, thescan may include a bit-mapped image via the optical capture device 708.Logic 726 causes the bit-mapped image to be stored in memory 730 with anassociated time-stamp read from the clock unit 728. Logic 726 may alsoperform optical character recognition (OCR) or other post-scanprocessing on the bit-mapped image to convert it to text. Logic 726 mayoptionally extract a signature from the image, for example by performinga convolution-like process to locate repeating occurrences ofcharacters, symbols or objects, and determine the distance or number ofother characters, symbols, or objects between these repeated elements.The reader may then upload the bit-mapped image (or text or othersignature, if post-scan processing has been performed by logic 726) toan associated computer via network interface 716.

As an example of another use of mobile device 702, a reader may capturesome text from an article as an audio file by using microphone 710 as anacoustic capture port. Logic 726 causes audio file to be stored inmemory 730. Logic 726 may also perform voice recognition or otherpost-scan processing on the audio file to convert it to text. As above,the reader may then upload the audio file (or text produced by post-scanprocessing performed by logic 726) to an associated computer via networkinterface 716.

A directional sensor 741 may also be incorporated into the mobile device702. The directional device may be a compass and be based upon amagnetic reading, or based upon network settings.

In the following sections, detailed descriptions of examples and methodsof the invention will be given. The description of both preferred andalternative examples though through are exemplary only, and it isunderstood that to those skilled in the art that variations,modifications and alterations may be apparent. It is therefore to beunderstood that the examples do not limit the broadness of the aspectsof the underlying invention as defined by the claims.

Referring now to FIG. 8, exemplary steps that may be performed in someaspects of the present invention are illustrated. At step 801, aprocessor may generate an AVM model of a Structure. The AVM model may bebased upon a physical layout of the Structure and include a layout ofeach item of machinery, equipment as well as facility features. At step802, the AVM may receive data indicative of one or more performancemetrics. Data may include data generated via a sensor and/or input by auser. In some examples, data may include performance metrics, utilitycost, maintenance cost and replacement cost.

At step 803, a data connection between a deployed facility and an AVMmay be automated to generate and transmit data to the model on anautomated basis without human intervention or artificial delay. All orsome data may be stored in a storage. At step 804, the AVM may accessreceived and/or historical data from the same or other AVM models. Atstep 805. Artificial Intelligence routines or other logic may integraterelevant indices, including one or more of: geographic location, labororganization, market conditions, labor costs, physical conditions,property status or data descriptive of other variables.

At step 806, an AVM may generate a value for build and deployment cost,and at step 807 the AVM may include utility and consumables cost. Atstep 808 an AVM may generate one or more of: predicted and actualquantifications from the structure; energy consumption and processthroughput.

Referring now to FIG. 9A, an exemplary perspective graph 900 comprisingthree separate perspective points 925, 945, 965 is illustrated. In someaspects, as illustrated in FIG. 9B, a wearable display 905 may beconfigured to detect eye movement of the wearer 915, which may becalibrated. For example, such as illustrated in FIG. 9B, a neutral,forward-looking eye position 920 may be established as the center pointof the axes 910 (0, 0), which may establish a view along the positivez-axis. As a further illustrative example in FIG. 9C, once calibrated, ashift in eye position 940 to look up and left may change a view from thevantage point and be transmitted to the AVM to access another portion ofthe AVM. As an illustrative example, as shown in FIG. 9D, a user maylook right, and the eye position 960 may shift along the positivex-axis.

In some aspects, the wearable display 905 may comprise a set of gogglesor glasses, wherein the goggles or glasses may comprise one or morelenses. For example, a single wrapped lens may allow a user toexperience panoramic views. Alternately, dual lenses may providedifferent image data, wherein the combined images may allow the user tohave stereoscopic perception of the performance event. In still furtherembodiments, the wearable display 905 may comprise a helmet, which mayallow for more detailed immersion. For example, a helmet may allow fortemperature control, audio isolation, broader perspectives, orcombinations thereof.

Referring now to FIGS. 10A-10C, exemplary horizontal changes in viewingareas are illustrated. In some embodiments, the wearable display maycomprise an accelerometer configured to detect head movement. Similarlyto the eye position detection, the accelerometer may be calibrated tothe natural head movements of a user 1000. In some embodiments, thecalibration may allow the user to tailor the range to the desiredviewing area. For example, a user may be able to move their head 110°comfortably, and the calibration may allow the user to view the entire180° relative the natural 110° movement.

As illustrated in FIG. 10A, a neutral head position 1020 of the wearabledisplay may allow the user 1000 to view a forward-looking perspective1025. As illustrated in FIG. 10B, a right head position 1040 of thewearable display may allow the user 1000 to view a rightward-lookingperspective 1045. As illustrated in FIG. 10C, a left head position 1060of the wearable display may allow the user 1000 to view aleftward-looking perspective 1065.

Referring now to FIGS. 11A-11C, exemplary vertical changes in viewingareas are illustrated. Similarly to FIGS. 10A-10C, in some embodiments,the wearable display may be configured to detect vertical motions. Insome aspects, a user may look up to shift the viewing area to a range inthe positive y axis grids, and user may look down to shift the viewingarea to a range in the negative y axis grids. In some embodiments, thewearable display may be configured to detect both horizontal andvertical head motion, wherein the user may be able to have almost a 270°viewing range.

As illustrated in FIG. 11A, a neutral head position 1120 of the wearabledisplay may allow the user 1100 to view a forward-looking perspective1125. As illustrated in FIG. 11B, an up head position 1140 of thewearable display may allow the user 1000 to view an upward-lookingperspective 1145. As illustrated in FIG. 11C, a down head position 1160of the wearable display may allow the user 1100 to view adownward-looking perspective 1165.

In still further embodiments, the wearable display may be able to detect360° of horizontal movement, wherein the user may completely turn aroundand change the neutral viewing range by 180°. In some aspects, thewearable display may be configured to detect whether the user may besitting or standing, which may shift the perspective and viewing area.In some implementations, a user may be allowed to activate or deactivatethe motion detection levels, based on preference and need. For example,a user may want to shift between sitting and standing throughout theexperience without a shift in perspective. In some implementations, thewearable display may further comprise speakers, wherein audio data maybe directed to the user.

In some embodiments, the wearable display may allow for immersion levelcontrol, wherein a user may adjust the level of light and transparencyof the wearable display and/or frames. In some aspects, the lenses ofthe wearable display may comprise an electrically active layer, whereinthe level of energy may control the opacity. For example, theelectrically active layer may comprise liquid crystal, wherein theenergy level may control the alignment of the liquid crystal. Where auser may prefer a fully immersive viewing experience, the lenses may beblacked out, wherein the user may see the video with minimal externalvisibility. Where a user may still prefer to have awareness orinteractions beyond the video, the lenses and/or frames may allow forsome light to penetrate or may allow for some transparency of the video.

Additional examples may include Sensor arrays, audio capture arrays andcamera arrays with multiple data collection angles that may be complete360 degree camera arrays or directional arrays, for example, in someexamples, a Sensor array (including image capture Sensors) may includeat least 120 degrees of data capture, additional examples include aSensor array with at least 180 degrees of image capture; and still otherexamples include a Sensor array with at least 270 degrees of imagecapture. In various examples, data capture may include Sensors arrangedto capture image data in directions that are planar or oblique inrelation to one another.

Referring now to FIG. 12, methods and devices for determining adirection that may be referenced for one or both of data capture and AVMpresentation of a particular portion of the virtual representation ofthe modeled structure. A User 1200 may position a Smart Device 1205 in afirst position 1201 proximate to a portion of a structure for which arepresentation in the AVM the User 1200 wishes to retrieve and display.The first position 1201 of the Smart Device 1205 may be determined (asdiscussed herein via GPS and/or triangulation) and recorded. The User1200 may then relocate the Smart Device 1205 to a second position 1202in a general direction of the portion of a structure (illustrated as theZ direction) for which a representation in the AVM the User 1200 wishesto retrieve and display. In this manner, the AVM system (not shown inFIG. 12) and/or the Smart Device 1205 may generate one or both of a rayand a vector towards the portion of a structure for which arepresentation in the AVM the User 1200 wishes to retrieve and display.

In some embodiments, the vector may have a length determined by the AVMthat is based upon a length of a next Feature in the AVM located in thedirection of the generated vector. The vector will represent a distance1203 from the second position 1202 to an item 1225 along the Z axisdefined by a line between the first position 1201 and the secondposition 1202. A ray will include a starting point and a direction.

As illustrated, the change in the Z direction is associated with a zerochange in the X and Y directions. The process may also include a secondposition 1205 that has a value other than zero in the X and/or Ydirections.

In other embodiments, a User 1200 may deploy a laser, accelerometer,sound generator or other device to determine a distance from the SmartDevice 1205 to the feature, such as a piece of equipment. Such uniquemethods of determining a location and direction of data capture may beutilized to gather data during construction of modeled buildings orother structures and during Deployment of the structures during theOperational Stage. An additional non-limiting example may includedirection based identification; with a fixed location, or in tandem witha location means, a device may have capabilities to deduce orientationbased information of the device. This orientation information may beused to deduce a direction that the device is pointing in. Thisdirection based information may be used to indicate that the device ispointing to a specific piece of equipment 1225 that may be identified inthe AVM.

In still other embodiments, a device with a controller and anaccelerometer, such as mobile Smart Device 1205, may include a userdisplay that allows a direction to be indicated by movement of thedevice from a determined location acting as a base position towards anAs Built feature in an extended position. In some implementations, theSmart Device determines a first position 1201 based upon triangulationwith the reference points. The process of determination of a positionbased upon triangulation with the reference points may be accomplished,for example via executable software interacting with the controller inthe Smart Device, such as, for example by running an app on the SmartDevices 1205.

In combination with, or in place of directional movement of a SmartDevice 1205 in order to quantify a direction of interest to a user, someembodiments may include an electronic and/or magnetic directionalindicator that may be aligned by a user in a direction of interest.Alignment may include, for example, pointing a specified side of adevice, or pointing an arrow or other symbol displayed upon a userinterface on the device towards a direction of interest.

In a similar fashion, triangulation may be utilized to determine arelative elevation of the Smart Device as compared to a referenceelevation of the reference points.

Other techniques for position determination, such as a fingerprinttechnique that utilizes a relative strength of a radio signal within astructure to determine a geospatial position. are also within the scopeof the present invention.

It should be noted that although a Smart Device is generally operated bya human user, some embodiments of the present invention include acontroller, accelerometer, and data storage medium, Image CaptureDevice, such as a Charge Coupled Device (“CCD”) capture device and/or aninfrared capture device being available in a handheld or unmannedvehicle.

An unmanned vehicle may include for example, an unmanned aerial vehicle(“UAV”) or ground level unit, such as a unit with wheels or tracks formobility and a radio control unit for communication.

In some embodiments, multiple unmanned vehicles may capture data in asynchronized fashion to add depth to the image capture and/or a threedimensional and 4 dimensional (over time) aspect to the captured data.In some implementations, UAV position will be contained within aperimeter and the perimeter will have multiple reference points to helpeach UAV (or other unmanned vehicle) determine a position in relation tostatic features of a building within which it is operating and also inrelation to other unmanned vehicles. Still other aspects includeunmanned vehicles that may not only capture data but also function toperform a task, such as paint a wall, drill a hole, cut along a definedpath, or other function. As stated throughout this disclosure, thecaptured data may be incorporated into an AVM.

In still other embodiments, captured data may be compared to a libraryof stored data using recognition software to ascertain and/or affirm aspecific location, elevation and direction of an image capture locationand proper alignment with the virtual model. Still other aspects mayinclude the use of a compass incorporated into a Smart Device.

By way of non-limiting example, functions of the methods and apparatuspresented herein may include one or more of the following factors thatmay be modeled and/or tracked over a defined period of time, such as,for example, an expected life of a build (such as, 10 years or 20years).

Referring now to FIG. 13, additional apparatus and methods fordetermining a geospatial location and determination of a direction ofinterest may include one or both of an enhanced smart device and a smartdevice in logical communication with wireless position devices1303-1310. The importance of geospatial location and determination of adirection of interest is discussed in considerable detail above. Asillustrated, a smart device 1301 may be in logical communication withone or more wireless position devices 1303-1310 strategically located inrelation to the physical dimensions of the smart device. For example,the smart device 1301 may include a smart phone or tablet device with auser interface surface 1320 that is generally planar. The user interfacesurface 1320 will include a forward edge 1318 and a trailing edge 1319.

In some preferred embodiments, the smart device will be fixedly attachedto a smart receptacle 1302. The smart receptacle 1302 may include anappearance of a passive case, such as the type typically used to protectthe smart device 1301 from a damaging impact. However, according to thepresent invention, the smart receptacle 1302 will include digital and/oranalog logical components, such as wireless position devices 1303-1310.The wireless position devices 1303-1310 include circuitry capable ofreceiving wireless transmissions from multiple wireless positionalreference transceivers 1311-1314. The wireless transmissions willinclude one or both of analog and digital data suitable for calculatinga distance from each respective reference point 1311-1314.

In some embodiments, the smart receptacle 1302 will include a connector1315 for creating an electrical path for carrying one or both ofelectrical power and logic signals between the smart device 1301 and thesmart receptacle 1302. For example, the connector 1315 may include amini-usb connector or a lightening connector. Additional embodiments mayinclude an inductive coil arrangement for transferring power.

Embodiments may also include wireless transmitters and receivers toprovide logical communication between the wireless position devices1303-1310 and the smart device 1301. Logical communication may beaccomplished, for example, via one or more of: Bluetooth, ANT, andinfrared mediums.

Reference transceivers 1311-1313 provide wireless transmissions of datathat may be received by wireless position devices 1303-1310. Thewireless transmissions are utilized to generate a position of therespective wireless position devices 1303-1310 in relation to theAccording to the present invention, Reference transceivers 1311-1313providing the wireless transmissions to the wireless position devices1303-1310 are associated with one or more of: a position in a virtualmodel; a geographic position; a geospatial position in a defined area,such as structure; and a geospatial position within a defined area (suchas, for example a real property).

According to the present invention, a smart device may be placed into acase, such as a smart receptacle 1302 that includes two or more wirelessposition devices 1303-1310. The wireless position devices 1303-1310 mayinclude, for example, one or both of: a receiver and a transmitter, inlogical communication with an antenna configured to communicate withreference transceivers 1311-1314. Communications relevant to locationdetermination may include, for example, one or more of: timing signals;SIM information; received signal strength; GPS data; raw radiomeasurements; Cell-ID; round trip time of a signal; phase; and angle ofreceived/transmitted signal; time of arrival of a signal; a timedifference of arrival; and other data useful in determining a location.

The wireless position devices 1303-1310 may be located strategically inthe case 1302 to provide intuitive direction to a user holding the case1302, and also to provide a most accurate determination of direction.Accordingly, a forward wireless position device 1303 may be placed at atop of a smart device case and a reward wireless position device 1304may be placed at a bottom of a smart device case 1302. Some embodimentseach of four corners of a case may include a wireless position device1305, 1306, 1307, 1308. Still other embodiments may include a wirelessposition device 1309 and 1310 on each lateral side.

The present invention provides for determination of a location of two ormore wireless positioning devices 1303-1310 and generation of one ormore directional vectors 1317 and/or rays based upon the relativeposition of the wireless positioning devices 1303-1310. For the sake ofconvenience in this specification, discussion of a vector that does notinclude specific limitations as to a length of the vector and isprimarily concerned with a direction, a ray of unlimited length may alsobe utilized. In some embodiments, multiple directional vectors 1317 aregenerated and a direction of one or more edges, such as a forward edge,is determined based upon the multiple directional vectors 1317.

According to the present invention a geospatial location relative to oneor more known reference points is generated. The geospatial location inspace may be referred to as having an XY position indicating a planardesignation (e.g. a position on a flat floor), and a Z position (e.g. alevel within a structure, such as a second floor) may be generated basedupon indicators of distance from reference points. Indicators ofdistance may include a comparison of timing signals received fromwireless references. A geospatial location may be generated relative tothe reference points. In some embodiments, a geospatial location withreference to a larger geographic area is associated with the referencepoints, however, in many embodiments, the controller will generate ageospatial location relative to the reference point(s) and it is notrelevant where the position is located in relation to a greatergeospatial area.

In some embodiments, a position of a smart device may be ascertained viaone or more of: triangulation; trilateration; and multilateration (MLT)techniques.

A geospatial location based upon triangulation may be generated basedupon a controller receiving a measurement of angles between the positionand known points at either end of a fixed baseline. A point of ageospatial location may be determined based upon generation of atriangle with one known side and two known angles.

A geospatial location based upon trilateration may be generated basedupon a controller receiving wireless indicators of distance and geometryof geometric shapes, such as circles, spheres, triangles and the like.

A geospatial location based upon multilateration may be generated basedcontroller receiving measurement of a difference in distance to tworeference positions, each reference position being associated with aknown location. Wireless signals may be available at one or more of:periodically, within determined timespans and continually. Thedetermination of the difference in distance between two referencepositions provides multiple potential locations at the determineddistance. A controller may be used to generate a plot of potentiallocations. In some embodiments, the potential determinations generallyform a curve. Specific embodiments will generate a hyperbolic curve.

The controller may be programmed to execute code to locate an exactposition along a generated curve, which is used to generate a geospatiallocation. The multilateration thereby receives as input multiplemeasurements of distance to reference points, wherein a secondmeasurement taken to a second set of stations (which may include onestation of a first set of stations) is used to generate a second curve.A point of intersection of the first curve and the second curve is usedto indicate a specific location.

In combination with, or in place of directional movement of a SmartDevice 1301 in order to quantify a direction of interest to a user, someembodiments may include an electronic and/or magnetic directionalindicator that may be aligned by a user in a direction of interest.Alignment may include, for example, pointing a specified side of adevice, or pointing an arrow or other symbol displayed upon a userinterface on the device towards a direction of interest.

In a similar fashion, triangulation may be utilized to determine arelative elevation of the Smart Device as compared to a referenceelevation of the reference points.

It should be noted that although a Smart Device is generally operated bya human user, some embodiments of the present invention include acontroller, accelerometer, and data storage medium, Image CaptureDevice, such as a Charge Coupled Device (“CCD”) capture device and/or aninfrared capture device being available in a handheld or unmannedvehicle.

An unmanned vehicle may include for example, an unmanned aerial vehicle(“UAV”) or an unmanned ground vehicle (“UGV”), such as a unit withwheels or tracks for mobility. A radio control unit may be used totransmit control signals to a UAV and/or a UGV. A radio control unit mayalso receive wireless communications from the unmanned vehicle.

In some embodiments, multiple unmanned vehicles may capture data in asynchronized fashion to add depth to the image capture and/or a threedimensional and 4 dimensional (over time) aspect to the captured data.In some implementations, a UAV position will be contained within aperimeter and the perimeter will have multiple reference points to helpeach UAV (or other unmanned vehicle) determine a position in relation tostatic features of a building within which it is operating and also inrelation to other unmanned vehicles. Still other aspects includeunmanned vehicles that may not only capture data but also function toperform a task, such as paint a wall, drill a hole, cut along a definedpath, or other function. As stated throughout this disclosure, thecaptured data may be incorporated into an AVM.

In still other embodiments, captured data may be compared to a libraryof stored data using recognition software to ascertain and/or affirm aspecific location, elevation and direction of an image capture locationand proper alignment with the virtual model. Still other aspects mayinclude the use of a compass incorporated into a Smart Device.

By way of non-limiting example, functions of the methods and apparatuspresented herein may include one or more of the following factors thatmay be modeled and/or tracked over a defined period of time, such as,for example, an expected life of a build (such as, 10 years or 20years).

Referring now to FIG. 13A, in some embodiments, wireless positiondevices 1303A-1310A may be incorporated into a smart device 1301A andnot require a smart receptacle to house wireless position devices1303-1310. Wireless position devices 1303A-1310A that are incorporatedinto a smart device, such as a smart phone or smart tablet, will includeinternal power and logic connections and therefore not require wirelesscommunication between the controller in the smart device 1301A and theC.

A person of ordinary skill in the arts will understand that a smartdevice 1301A with integrated wireless position devices 1303-1310 and asmart device 1301 with wireless position devices 1303-1310 in a smartreceptacle 1302 may provide a directional indication, such as adirectional vector 1317 1317A, without needing to move the smart devicefrom a first position to a second position since a directional vectormay be determined from a relative position of a first wireless positiondevices 1303-1310 and a second wireless positional device wirelessposition devices 1303-1310.

In exemplary embodiments, as described herein, the distances may betriangulated based on measurements of WiFi strength at two points. WiFisignal propagates outward as a wave, ideally according to an inversesquare law. Ultimately, the crucial feature of the present inventionrelies on measuring relative distances at two points. In light of thespeed of WiFi waves and real-time computations involved in orienteering,these computations need to be as computationally simple as possible.Thus, depending upon the specific application and means for taking themeasurements, various coordinate systems may be desirable. Inparticular, if the smart device moves only in a planar direction whilethe elevation is constant, or only at an angle relative to the ground,the computation will be simpler.

Accordingly, an exemplary coordinate system is a polar coordinatesystem. One example of a three-dimensional polar coordinate system is aspherical coordinate system. A spherical coordinate system typicallycomprises three coordinates: a radial coordinate, a polar angle, and anazimuthal angle (r, θ, and φ, respectively, though a person of ordinaryskill in the art will understand that θ and φ are occasionally swapped).

By way of non-limiting example, suppose Point 1 is considered the originfor a spherical coordinate system (i.e., the point (0, 0, 0)). Each WiFiemitter e₁, e₂, e₃ can be described as points (r₁, θ₁, φ₁), (r₂, θ₂,φ₂), and (r₃, θ₃, φ₃), respectively. Each of the re's (1≤i≤3) representthe distance between the WiFi emitter and the WiFi receiver on the smartdevice.

In some embodiments, the orienteering occurs in a multi-story building,in which WiFi emitters may be located above and/or below the technician.In these embodiments, a cylindrical coordinate system may be moreappropriate. A cylindrical coordinate system typically comprises threecoordinates: a radial coordinate, an angular coordinate, and anelevation (r, θ, and z, respectively). A cylindrical coordinate systemmay be desirable where, for example, all WiFi emitters have the sameelevation.

Referring now to FIG. 13B, in some embodiments, one or both of a smartdevice 1301 and a smart receptacle 1302 may be rotated in a manner (suchas, for example in a clockwise or counterclockwise movement 1320 1322relative to a display screen) that repositions one or more wirelessposition devices 1303-1310 from a first position to a second position. Avector 1326 may be generated at an angle that is perpendicular 1325 orsome other designated angle in relation to the smart device 1301. Insome embodiments, an angle in relation to the smart device isperpendicular 1325 and thereby viewable via a forward looking camera onthe smart device.

A user may position the smart device 1301 such that an object in adirection of interest is within in the camera view. The smart device maythen be moved to reposition one or more of the wireless position devices1303-1310 from a first position to a second position and thereby capturethe direction of interest via a generation of a vector in the directionof interest.

Referring now to FIG. 13C, as illustrated, a vector in a direction ofinterest 1325 may be based upon a rocking motion 1323-1324 of the smartdevice 1301, such as a movement of an upper edge 1318 in a forwardarcuate movement 1323. The lower edge 1319 may also be moved in acomplementary arcuate movement 1324 or remain stationary. The movementof one or both the upper edge 1318-1319 also results in movement of oneor more wireless position devices 1303-1310. The movement of thewireless position devices 1303-1310 will be a sufficient distance toregister to geospatial positions based upon wireless transmissions. Arequired distance will be contingent upon a type of wirelesstransmission referenced to calculate the movement, For example, aninfrared beam may require less distance than a WiFi signal, and a WiFitransmission may require less distance than a cell tower transmissionwhich in turn may require less distance than a GPS signal.

Referring now to FIG. 14, in still other embodiments, a smart device1415 may be logically associated with a larger platform 1400 forsupporting wireless position devices 1401-1412. The larger platform 1400may include a vehicle, such as an automobile, a truck, a ship, anaircraft, a motorcycle or other motorized vehicle. As illustrated theplatform 1400 includes an automobile. The platform 1400 may includealmost any combination of two or more wireless position devices1401-1412 that may provide respective positional data sufficient togenerate a directional vector. Accordingly, by way of non-limitingexample, a front and center wireless position device 1401 may be pairedwith a rear center wireless position device 1402; each corner of thevehicle may include a wireless position device 1403-1406; interiorcorners may include a respective wireless position device 1409-1412; andexterior locations, such as on rear view mirrors may contain wirelessposition devices 1407-1408.

Utilizing multiple on board wireless position devices 1401-1412, it ispossible to ascertain a direction that a vehicle is pointing withoutmovement of the vehicle. This is useful since unlike traditional methodsutilized by navigational systems that relied on a first geographiclocation of the vehicle and a second geographic position of the vehicle,which in turn required motion, the present invention provides fordirectional orientation without movement of the vehicle.

In another aspect, a controller may be included in a smart device pairedto the vehicle and/or a transmitter 1416 may transmit data received fromthe multiple wireless position devices 1401-1412 to a remote processorwhich may determine a directional orientation. The remote processorand/or a smart device may also transmit the directional orientation backto a display viewable by an operator of the vehicle.

Referring now to FIGS. 15A-15C, a support 1500 for a smart device 1501is illustrated. The support remains stationary in relation to a groundplane. One or more position devices 1503-1508 are shown located within,on or proximate to the smart device 1501. In FIG. 15A, generally linearmovement 1514-1515 from a first position to a second position isillustrated. In some embodiments, a cessation of movement in a generaldirection is determined via an accelerometer included in or operated bythe smart device 1501. In other embodiments (show here as the support1500) may activate a user interactive device, such as a button on atouch screen, or a switch to indicate one or both of the first positionand the second position.

The wireless position devices 1503-1508 enter into logical communicationwith multiple wireless positional reference transceivers 1510-1513.

In some embodiments, a direction of interest will include an item ofinterest 1509, such as an apparatus or other piece of equipment. Adirection of interest 1514 may include a vector with a directionpointing towards the item of interest 1509. The vector length will besufficient to reach the item of interest 1509.

In some embodiments, a vector indicating a direction of interest 1514may be used to reference an AVM and the SVM may provide a selectionmechanism, such as a drop down menu that includes potential items ofinterest 1509 along the vector direction. A selection of an item ofinterest may then be used to determine a length of the vector 1514.

Referring now to FIG. 15C, a movement of a smart device 1501 may bearcuate in nature 1514C so long as arcuate movement 1514C results insufficient distance of movement of one or more position devices1503-1508.

Referring now to FIG. 16, method steps that may be implemented in someembodiments of the present invention are illustrated. At method step1600, geospatial location services are used to determine geospatiallocation such as a location of the structure with a position anddirection of interest. Geospatial services may be used to determine auser's location relative to the structure and directions thereto. Themethods used may include, by way of non-limiting example, one or moreof: satellite-based global positioning systems (GPS), cell towertriangulation, radio signal triangulation, Wi-Fi signal locationservices, infrared transmitters and the like.

Geospatial location services will be cross-referenced with databaseregistry of as built virtually modeled facilities and may be used inconjunction with a network of registered service technicians to routethe nearest available service technician to the structure experiencingequipment malfunction. Service technician may register with the systemto accept geospatial location tracking services by the system.

At method step 1601, the service technician's entry into the structurewill be registered. Registration of entry into the structure may beachieved through multiple methods, which may include, by way ofnon-limiting example, on or more of: WiFi gateway detection, infrareddetection, magnetic door locking systems, Bluetooth services, and thelike. Upon entry into the structure requesting the service call, systemwill register the service technician's entry into the structure.

At method step 1602, a support unit for a smart device, such as servicetechnician or an unmanned vehicle may be tacked via a change intriangulation values and/or an accelerometer and a position anddirection within the structure is tracked. The methods used may be, bymeans of non-limiting example, one or more of: use of data gleaned fromaccelerometers located on or in possession of service technicians, Wifiservices, radio frequency (RF) triangulation, Bluetooth technology,infrared detection, RFID badges, and the like.

At method step 1603, a smart device will be registered as enteringwithin structure. Following the smart device entry into structure.

At method step 1604, a smart device may be associated with one or bothof a person and an entity.

At method step 1605, the smart device is pre-registered by the systemwith detailed instructions regarding a reason for the device to be at aparticular location. The reason may be, for example, one or more of: aservice call placed from structure to system detailing current equipmentmalfunction, service calls from structure detailing non-specificmalfunctions and symptomatic data indicating equipment malfunction, aservice call placed by self-assessing equipment utilizing internet ofthings (IoT) and machine learning functionality to ascertainmalfunctions and predictive analytics to anticipate malfunctions, andthe like. The system may integrate data reports into the AVM and relayas much to the smart device in the field.

Alternatively, at method step 1605A, the smart device may arrive at thestructure without prior knowledge of a purpose. Upon entry into thestructure and registration of the smart device as described in methodsteps 1601 through 1604, system will relay data gleaned from the AVM,operational data uploaded to the system through IoT processes, and otherexperiential data reported to the system and thereby relayed to thesmart device on site. Methods for relation of such data to the on sitesmart device may include, by means of non-limiting example, referentialdata based on proprietary orienteering processes to determine smartdevice location within structure, which location will becross-referenced with AVM data.

At method step 1606, a position within or proximate to the structure maybe determined via positioning identifiers. The position within orproximate to the structure is determined and detailed instructionsdirecting smart device to the source of a malfunction is relayed by thesystem to the smart device directly or by means of smart deviceapplication. The methods used may be, by means of non-limiting example,one or more of: augmented reality overlays displayed on heads-updisplays or other wearable technologies, augmented reality overlaysdisplayed on smart devices, direct instructional vectoring provided tothe smart device by the system over Wifi internet connection or LTEsignal, virtual reality walkthrough instructions provided to smartdevice on site or prior to arrival at the structure, updatedmap/schematic displays detailing the structure and directing the smartdevice to the source of the subject malfunction by means of vectoringand orienteering processes.

At method step 1607, a smart device's location within the structurealong an XY axis will be tracked and recorded by the system by means offixed or adaptive orienteering apparatus within the structure. Suchorienteering apparatus may include, by means of non-limiting example, onor more of: WiFi triangulation, infrared position detection, radiofrequency (RF) detection, RFID tracking, onboard accelerometers locatedon the smart device or carried smart devices, and the like.

At method step 16016, the smart device's location within the structurealong the Z axis will be determined. The methods used may be, by meansof non-limiting example, one or more of: onboard magnetometers, onboardbarometers, onboard accelerometers, and the like, used in conjunctionwith in-structure XY axis position processes described in method step0167 above, along with data detailed in the AVM of the structure.

At method step 1609, the smart device's direction of interest will bedetermined. Method steps 1601 through 16016 work in conjunction to trackand direct the smart device to the source of the malfunction; once atthe source of the malfunction, smart device will be oriented to thedirection of interest. The system will determine the smart device'sdirection of interest using, by means of non-limiting example, on ormore of the following methods: infrared pointers, laser directionfinding devices, onboard camera(s), RFID trackers, RFD finders, barcodescanners, hex/hash code scanners, Wifi triangulation, and the like.

At method step 1610, the smart device's distance to the subjectmalfunction will be determined. The methods used may be, by means ofnon-limiting example, one or more of the following: infrared pointers,laser pointing devices, Wifi triangulation, RF ID sensors, RFD, depthperception sensors contained within onboard cameras, onboardmagnetometers, Bluetooth technology, ANT sensors, directionally enabledsmart device cases, and the like.

At method step 1611, records of equipment and/or area of interest willbe accessed and relayed to smart device. The smart device's position,direction of interest, and distance to the equipment/area of interest asdetermined by method steps 1601 through 1610 will be cross-referencedwith the AVM and experiential data to call up pertinent data on themalfunctioning equipment/area of interest. Data regarding the servicecall will be added to the AVM and experiential data displayed to theon-site smart device. The methods used may be, by means of non-limitingexample, one or more of: IoT data relayed by machine learning-enabledequipment, structure-relayed symptomatic data, and the like.

At method step 1612, symptomatic malfunction data will be diagnosed todetermine cause of malfunction. The methods used may be, by means ofnon-limiting example, one or more of: IoT experiential data gathered andcollated from multiple sources across multiple facilities similar to thepresented symptomatic data, internet-gathered data analyzed by variousmachine learning technologies, algorithmic analytics of symptomatic datato determine causal indications, and smart device expertise.

At method step 1613, technical maintenance data, records, andinstructional walkthrough data will be relayed to smart device. Systemwill collate data from method step 1612 above and relay as much to smartdevice. The methods used may be, by means of non-limiting example, oneor more of: augmented reality overlays as displayed by heads-up displaysand other wearable technologies, augmented reality overlays as displayedon smart devices, virtual reality walkthroughs as shown by wearabletechnologies or smart devices, direct instruction or remote control,.pdf user manuals and other written instructional material, videowalkthrough instructions displayed on smart devices, and the like.

At method step 1614, results of purpose for a presence at a location arerecorded and added as experiential data to the AVM. The methods used maybe, by means of non-limiting example, on or more of: equipmentself-reporting through IoT and machine learning technologies, smartdevice entered data, experiential data gathered from emplaced sensorsand other recording devices within the structure itself, and the like.

Referring now to FIG. 17A, methods steps that may be executed in someembodiments of the present invention are presented. At step 1701, asdiscussed in detail herein, Transceivers may be affixed to referencepositions within or proximate to a structure. In some preferredembodiment's Transceivers are positioned at the perimeter of thestructure and are capable of wireless communication form any pointwithin the structure.

At step 1702, a sensor is deployed to a position within, or proximateto, the structure in a manner conducive for the sensor to operate andgenerate data descriptive of a condition that is one or both of: withinthe structure or proximate to the structure. The sensor will alsogenerate a digital signal descriptive of the condition monitored by thesensor.

At step 1703, the sensor is activated to an operational state andgenerates the digital data and transmits the data. In some preferredembodiments, the sensor will transmit the data via a wirelesstransmission. In other embodiments, the sensor may transmit the data viaan electrical or optical connector.

At step 1704, a physical position of the sensor is determined based uponwireless communication of the sensor with two or more of the wirelesstransceivers at the reference positions. As discussed herein thephysical position may include an X coordinate and a Y coordinate on anX, Y plane and a elevation based upon a Z coordinate relative to aground plane or other designated plane of origin.

At step 1705 a digital signal is transmitted descriptive of thecondition of the structure. The condition of the structure may be basedupon Vital Conditions of the structure assessed via the sensor readings.At 1706 a physical state of the building at the physical position of thesensor, or an area of the structure within range of the sensor, may becorrelated with the digital signal descriptive of a condition of thestructure.

At step 1707, the sensor locations and/or areas of the structure withinrange of the sensor for which the sensor may take a reading, areassociated with location coordinates, such as X, Y and Z coordinates.

A step 1708, in another aspect, a direction of the sensor from a homeposition may be determined via the processes described herein. At step1709 a distance of sensor to an item of equipment or an area of interestmay be determined. The distance may be determined via methods, such asLIDAR, echo, User entry or other method.

At step 1710, an index may be activated based upon a time at which asensor was activated. The index may be used to create a chronologicalsequence of sensor readings. The index may also be used to synchronizemultiple sensor readings and thereby capture a holistic picture of astructure during a given time period.

The logical communication may include wireless communications via anindustrial scientific and medical (ISM) band wavelength which mayinclude wavelengths between 6.765 MHz and 246 GHz. WiFi is one protocolthat may be implemented for wireless communication, as is Bluetooth,ANT, infrared or other protocol.

Referring now to FIG. 18, illustrates exemplary structures that may beused to correlate sensor readings and generate an alert based uponvalues of conditions measured in a structure. The table may includeBuilding Vital. Sensor Readings 1801 and include unique identifiers ofparticular sensors 1802; a location of each sensor 1802; a time and datestamp of a sensor reading 1804; a value of the reading 1805 and an alertstatus, such as one of: Normal, High, Low or almost any other statusindicator.

Another exemplary table may include Building Ratings Based Upon VitalReadings 1807. The Building Ratings table may include a PropertyIdentification 1808, such as a taxmap number or other identifier; anidentification of sensors deployed 1809, and aggregate status of theproperty as a whole 1810; a deployment impact 1811 that indicateswhether sensor readings correlate with acceptable readings for aspecified deployment; and an overall Property Rating 1812 which may be aresult of conditional logic applied to sensor readings of the sensorsdeployed.

Referring now to FIG. 19 as discussed further herein, a sensor thatincludes a microelectromechanical system (MEMS) accelerometer may beused to track vibration patterns. In some embodiments, a MEMSaccelerometer 1905 may be included within a smart device, such as atablet or a smart phone 1904. Other embodiments include a sensorindependent of a smart device. Still other embodiments include a sensorpackaged with a controller for executing software specific to thesensor, such as the Fluke™ 3561 FC Vibration Sensor. A structuralcomponent 1901 of a structure for which conditions will be monitoredwith sensors may include a vibration integrator 1902 with an attachmentfixture 1903 that establishes vibrational integrity between anaccelerometer 1905 in a smart phone 1904 and the structural component1901. The vibration integrator 1902 may be matched via its shape andmaterial to accurately convey vibrations present in the structuralcomponent to the accelerometer 1905 in the smart device 1904. In someembodiments a vibration integrator may include a damper or filter toexclude certain frequencies that may be considered noise to someapplications. A damper may be directional such that only vibrationfrequencies in a particular direction are excluded.

It is understood that an accelerometer 1905 does not need to beincorporated into a smart phone and may be directly fixed to anattachment fixture 1903 or fixed to a vibration integrator 1902 or fixedto a structural component 1901.

Vibrations present in the structural component may be indicative of astate of functioning of equipment included in the structure (not shownin FIG. 19). For example a first pattern of vibrations, which mayinclude frequency and/or amplitude and variations of one or both offrequency and amplitude may indicate a proper functioning of a piece ofequipment. Patterns of equipment installed in a setting in a structuremay be recorded under proper operating conditions to set up an initialproper state of functioning. Patterns derived from a subsequent sensorreading, such as an accelerometer 1905 reading may indicate a variationfrom the initial pattern of sufficient magnitude to indicate amalfunction or wear present in the equipment.

In some embodiments, a user, such as a service technician, may installan accelerometer into the attachment fixture for the specific purpose oftaking an accelerometer reading. A smart phone 1904 may run an app thatrecords a time and place and vibration pattern received. The vibrationpattern may be compared with a known set of vibration patterns and acondition of the structured may be ascertained from the comparison. Thetime date and vibration pattern may be transmitted to a server andaggregated with other sensor readings.

In another aspect, in some embodiments, a second accelerometer 1905A maybe used to introduce a vibration pattern into the structural component1901. The vibration pattern introduced may include a known frequency andamplitude. In some embodiments, the vibration pattern will include asequence of frequencies and amplitudes, wherein different frequenciesand amplitudes will be effective in diagnosing or otherwise indicatingan underlying causation for a pattern of vibration. The secondaccelerometer 1905A and the first accelerometer 1905 may be synchronizedvia executable software such that the first accelerometer will detectthe vibrations introduced by the second accelerometer 1905A. Anydiscrepancies between what was introduced by the first accelerometer1905A and the first accelerometer 1905 may be indicative of a state ofthe structure.

For example, introduction of a frequency pattern into a beam that issound may transmit well through the beam and be detected with minimalvariations from the frequency pattern that was introduced. However, abeam that is cracked or has rot within it may not convey the frequencypattern to the first accelerometer or convey the frequency pattern withsignificant distortion and/or diminutions in amplitude.

A history of sensor readings associated with a particular structureand/or group of structures may be stored and referenced to assist ininterpreting a cause for a particular vibration pattern.

Vibration sensors may be installed and recorded in as built data, oradded to a structure in a retrofit. Some commercial sensors (such as theFluke 3561 FC Vibration Sensor) may be associated with vendor suppliedsoftware for ease of retrofit implementation.

According to the present invention, accelerometers or other vibrationsensors are deployed in specific locations and tracked in a structureaccording to the respective sensor location. In addition, a relativeposition of a particular sensor position is tracked relative to othersensors (vibration sensors or sensors for monitoring differentmodalities of ambient conditions). The present system includes an AVMthat may store and make available to a user and/or to AI applicationswhich structural components are in vibrational communication with aparticular sensor. Various sensors include underlying piezoelectric,accelerometers of other technologies.

Embodiments also include a sensor programmed to reside in a lower powerstates and to periodically “wake itself up” (enter a higher poweredstate) to take a reading and transmit the reading. Sensor readings maybe correlated with different types of wear, damage, failure or properoperation of components included in a structure. The AVM tracks locationand may rank a likelihood of a component responsible for a particularvibration pattern detected by a sensor. The ranking may be based uponproximity, mediums available for communicating the vibration pattern(such as a beam traversing a significant portion of a structure butwhich provides excellent mechanical communication for the vibration).

Some embodiments also associate a sensor reading of vibration with atype of motion likely to cause such a reading. For example, somereadings may include a linear component and a rotational component (suchas operation of a washing machine during certain cycles). Patterns ofnormal and abnormal operation may be recorded and deciphered viaprogrammable software on a controller.

In another aspect, a pattern of sensor data that denotes spikes oflinear data may be associated with a human being walking. Overtime, acontroller may track sensor reading patterns and associate a particularpattern with the walk of a particular person.

It is also within the scope of the invention to track and analyze a setof data associated with a primary signal and additional sets of data(secondary, tertiary etc.) tracking harmonics of the primary signal. TheAVM may also track sets of data associated with simultaneous, and/orclosely timed readings received from multiple sensors and associate anamplitude, sequence, delay or other attribute of the data sets relativeto each other to provide input as to a location of a source of thevibration. Additionally, vibration sensors may include axis within thesensor. For example, two axis and three axis sensors may have adirection of each axis included in the AVM and used in analysis of avibration pattern.

The present invention also provides simple and fast procedures for theprovision of directions of a User or a sensor to a source of vibrationbased upon analysis of readings of one or more sensors via the X. Y andZ location determination and directional ray or vector generationmethods described herein.

Disparate types of sensor may also provide disparate data types that areuseful in combination to determine a source of sensor readings. Forexample, a vibration sensor reading indicating erratic motion may becombined with an increased temperature reading from a sensor proximateto an item of equipment. The combined sensor readings may assist in ananalysis of a cause of the sensor readings.

In still another aspect, one or more sensor readings may be correlatedto a life expectancy of an item of equipment, such as for example aheating Ventilation and Air Conditioning (HVAC) unit. By way ofnon-limiting example, an ammeter sensor reading measuring an electricaldraw of an HVAC unit may be quantified upon deployment of the unit. Theinitial readings may act as a baseline of a unit in excellentoperational condition. A similar baseline reading may be taken via anaccelerometer measuring a vibration generated by the HVAC unit. Stillother sensor readings may include airflow, temperature, humidity, orother condition. Over time, a change in one or more senor reading valuesmay indicate some wear and move the HVAC equipment item into a “normalwear but operational” status.

Still further along a time continuum, one or more sensor readings mayindicate a pending failure. For example, a current required to run theunit may be measured by the ammeter sensor and indicate an increaseddraw in electrical current. Likewise, airflow may decrease, andtemperature may increase, and other sensors may provide additionalevidence of a pending failure. Finally, a failed unit may generate avery high temperature reading and ammeter readings may increase to alevel of sufficient electrical current draw to trip an electricalbreaker, thereby indicating a failure.

According to the present invention, any of the sensor readings (or all,or some subset of all sensor readings) may be referenced to generate analert. Following the alert, remedial action may be taken.

Referring now to FIG. 17B, method steps for utilizing accelerometersand/or other sensors are presented. At step 1711 a vibration isintroduced into a component of the structure and at step 1712 thevibration pattern introduced is compared with a pattern of vibrationdetected by a MEMS. At step 1713 a series of transitions of MEMSaccelerometer readings may be tracked and at step 1714 the pattern ofvibrations measured by the MEMS accelerometers may be correlated withstructural integrity.

At step 1715, alternatively, structural damage may be correlated with apattern of vibration measured. At step 1716, a threshold range of valuesmeasured by a sensor may be set and at step 1717 an alert routine may beexecuted in the event that the threshold range is exceeded or upon ananalysis that detects a particular signal indicating a condition in thestructure.

Particular embodiments of the subject matter have been described. Otherembodiments are within the scope of the following claims. In some cases,the actions recited in the claims can be performed in a different orderand still achieve desirable results. In addition, the processes depictedin the accompanying figures do not necessarily require the particularorder show, or sequential order, to achieve desirable results. Incertain implementations, multitasking and parallel processing may beadvantageous. Nevertheless, it will be understood that variousmodifications may be made without departing from the spirit and scope ofthe claimed invention.

The invention claimed is:
 1. A method of assessing conditions in astructure in the presence of a user, the method comprising: a) affixingat least first, second, and third wireless transceivers to at leastfirst, second, and third reference positions, respectively, in orproximate to the structure, wherein the reference positions refer toestablished locations; b) deploying a sensor at a sensor location fromwhich the sensor is capable of generating digital data descriptive of acondition present in the structure, wherein the sensor is in wirelesscommunication with at least the first wireless transceiver; c)activating the sensor to generate the digital data descriptive of thecondition present in the structure; d) determining a physical positionof the sensor based upon wireless communication between at least two ofthe first, second, and third transceivers and a transceiver located atthe physical location of the sensor; e) associating a smart devicecomprising a tag with a user; f) placing the smart device in logicalcommunication with at least a first wireless position device and asecond wireless position device, each of the at least first wirelessposition device and the second wireless position device located withinor proximate to the smart device and capable of receiving wirelesstransmissions from the at least first, second, and third wirelesstransceivers; and g) with a controller, determining X, Y and Zcoordinates of the one or both of: the first wireless position deviceand the second wireless position device based, at least in part, on thewireless communication of the at least first and/or second wirelessposition device with the at least first, second, and third wirelesstransceivers; h) tracking and recording the user's position in thestructure by repeating step (g) over time; i) transmitting the digitaldata descriptive of the condition present from the sensor to thecontroller; j) receiving from the user a direction of interest, whereinan indication of the direction of interest is obtained by wirelesscommunication between the first wireless position device and the secondwireless position device with the least first, second and third wirelesstransceivers, and the direction is determined from the position of theposition of the first wireless position device to the position of thesecond wireless position device; and k) generating a user interfaceindicating a record of the user's presence in the structure and a recordof the condition present in the structure while the user is in thestructure.
 2. The method of claim 1, wherein the sensor detects at leastone condition selected from the group consisting of: humidity,particulates, vibration, chemical presence, gas presence, light,electromagnetic radiation, electrical power, moisture and mineralpresence.
 3. The method of claim 1, wherein the method further comprisesat least one of the at least first, second, or third referencetransceivers notifying the controller when the smart device enters thestructure.
 4. The method of claim 1, further comprising the smartdevice: l) correlating the digital data descriptive of the conditionpresent in the structure with a physical state of the structure; and m)providing an indication on the user interface of the physical state ofthe structure when the user's smart device is present in the structure.5. The method of claim 1, further comprising the step of generating anindex entry based upon a time at which the sensor was activated togenerate the digital data descriptive of the condition present in thestructure.
 6. The method of claim 1, further comprising the step oftransmitting the digital data via a wireless communication protocol overan industrial scientific and medical (ISM) band wavelength.
 7. Themethod of claim 6, wherein the wireless communication protocol comprisesa WiFi protocol.
 8. The method of claim 6, wherein the wirelesscommunication protocol comprises an infrared transmission.
 9. The methodof claim 6, wherein the wireless communication protocol comprises aBluetooth transmission.
 10. The method of claim 1, comprising repeatingsteps b) through k) for multiple respective sensors and additionallycomprising the step of synchronizing the digital data descriptive of thecondition present in the structure based upon a time at which themultiple respective sensors are activated to generate the digital data.11. The method of claim 1, wherein the condition present in thestructure comprises an amount of vibration present and the sensorscomprise microelectromechanical system (MEMS) accelerometers.
 12. Themethod of claim 1, further comprising the steps of setting a thresholdrange of values for the sensor reading and executing an alert routinebased on the sensor generating the digital data descriptive of thecondition present in the structure that is outside the threshold range.13. The method of claim 1, wherein the sensor detects a conditioncomprising temperature.
 14. The method of claim 13, wherein the sensoris co-located with the smart device comprising a tag.